[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480524886 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002311] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007598] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011134] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013031] SELinux: Initializing. [ 0.015067] *** VALIDATE selinux *** [ 0.021751] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025748] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027096] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029111] *** VALIDATE tmpfs *** [ 0.031080] *** VALIDATE proc *** [ 0.032224] *** VALIDATE cgroup *** [ 0.033000] *** VALIDATE cgroup2 *** [ 0.033232] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036028] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.039950] debug: unmapping init [mem 0xffffffffaca59000-0xffffffffaca60fff] [ 0.041153] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042636] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043020] ... version: 2 [ 0.044010] ... bit width: 48 [ 0.045011] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047012] ... max period: 00007fffffffffff [ 0.048015] ... fixed-purpose events: 3 [ 0.049010] ... event mask: 000000070000000f [ 0.050282] rcu: Hierarchical SRCU implementation. [ 0.052426] smp: Bringing up secondary CPUs ... [ 0.053536] x86: Booting SMP configuration: [ 0.054020] .... node #0, CPUs: #1 #2 #3 [ 0.058278] smp: Brought up 1 node, 4 CPUs [ 0.060013] smpboot: Max logical packages: 1 [ 0.061009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144383] node 0 deferred pages initialised in 81ms [ 0.147309] devtmpfs: initialized [ 0.149256] x86/mm: Memory block size: 128MB [ 0.152475] gcov: version magic: 0x41383552 [ 0.156273] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162314] pinctrl core: initialized pinctrl subsystem [ 0.165161] [ 0.165789] ************************************************************* [ 0.168014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171012] ** ** [ 0.174011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.176010] ** ** [ 0.178012] ** This means that this kernel is built to expose internal ** [ 0.181010] ** IOMMU data structures, which may compromise security on ** [ 0.182009] ** your system. ** [ 0.185010] ** ** [ 0.188016] ** If you see this message and you are not debugging the ** [ 0.191013] ** kernel, report this immediately to your vendor! ** [ 0.193009] ** ** [ 0.195011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198011] ************************************************************* [ 0.200816] NET: Registered protocol family 16 [ 0.202496] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.205052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.207098] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211011] cpuidle: using governor menu [ 0.214096] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.215515] PCI: Using configuration type 1 for base access [ 0.217127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.230056] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.232013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.240053] cryptd: max_cpu_qlen set to 1000 [ 0.244258] ACPI: Added _OSI(Module Device) [ 0.246009] ACPI: Added _OSI(Processor Device) [ 0.247010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.248024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.252829] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.259392] ACPI: Interpreter enabled [ 0.261056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.262012] ACPI: Using IOAPIC for interrupt routing [ 0.264124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.267355] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.277302] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.279029] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.282016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.285073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.289268] acpiphp: Slot [2] registered [ 0.291083] acpiphp: Slot [5] registered [ 0.292082] acpiphp: Slot [6] registered [ 0.293080] acpiphp: Slot [7] registered [ 0.294092] acpiphp: Slot [8] registered [ 0.296095] acpiphp: Slot [9] registered [ 0.297103] acpiphp: Slot [10] registered [ 0.299092] acpiphp: Slot [3] registered [ 0.301104] acpiphp: Slot [4] registered [ 0.302110] acpiphp: Slot [11] registered [ 0.303071] acpiphp: Slot [12] registered [ 0.304089] acpiphp: Slot [13] registered [ 0.305083] acpiphp: Slot [14] registered [ 0.306075] acpiphp: Slot [15] registered [ 0.308075] acpiphp: Slot [16] registered [ 0.309072] acpiphp: Slot [17] registered [ 0.310079] acpiphp: Slot [18] registered [ 0.311082] acpiphp: Slot [19] registered [ 0.312099] acpiphp: Slot [20] registered [ 0.313071] acpiphp: Slot [21] registered [ 0.315121] acpiphp: Slot [22] registered [ 0.316067] acpiphp: Slot [23] registered [ 0.317068] acpiphp: Slot [24] registered [ 0.318069] acpiphp: Slot [25] registered [ 0.319097] acpiphp: Slot [26] registered [ 0.321071] acpiphp: Slot [27] registered [ 0.322067] acpiphp: Slot [28] registered [ 0.323069] acpiphp: Slot [29] registered [ 0.324132] acpiphp: Slot [30] registered [ 0.325072] acpiphp: Slot [31] registered [ 0.327068] PCI host bridge to bus 0000:00 [ 0.328014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.330017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.332022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.334019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.336018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.338023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.339152] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.341864] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345280] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.354771] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.360053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.363020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.366017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.372620] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.375037] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.378040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.381565] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.386013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.399016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.403018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.409077] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.427014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.449019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.498023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.521359] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.541014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.557015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.578015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.599236] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.611016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.623017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.658017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.668720] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.677058] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.693015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.738017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.757434] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.785021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.797017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.843016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.862209] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.899024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.925025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.972039] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.008087] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.009367] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.011385] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.013355] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.015169] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.019204] iommu: Default domain type: Passthrough [ 1.021449] SCSI subsystem initialized [ 1.022134] ACPI: bus type USB registered [ 1.024099] usbcore: registered new interface driver usbfs [ 1.025054] usbcore: registered new interface driver hub [ 1.027145] usbcore: registered new device driver usb [ 1.028144] pps_core: LinuxPPS API ver. 1 registered [ 1.030014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.032053] PTP clock support registered [ 1.035042] EDAC MC: Ver: 3.0.0 [ 1.037137] PCI: Using ACPI for IRQ routing [ 1.038517] NetLabel: Initializing [ 1.040013] NetLabel: domain hash size = 128 [ 1.041007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.042075] NetLabel: unlabeled traffic allowed by default [ 1.045090] vgaarb: loaded [ 1.046226] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.047000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.053339] clocksource: Switched to clocksource kvm-clock [ 1.155410] VFS: Disk quotas dquot_6.6.0 [ 1.156747] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.158719] *** VALIDATE ramfs *** [ 1.159743] *** VALIDATE hugetlbfs *** [ 1.160953] pnp: PnP ACPI init [ 1.163045] pnp: PnP ACPI: found 6 devices [ 1.250870] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.254674] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.257080] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.259397] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.261397] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.263026] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.265911] NET: Registered protocol family 2 [ 1.268663] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.273901] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.277501] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.282931] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.286446] TCP: Hash tables configured (established 65536 bind 65536) [ 1.288987] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.292057] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.294778] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.297662] NET: Registered protocol family 1 [ 1.300558] RPC: Registered named UNIX socket transport module. [ 1.302804] RPC: Registered udp transport module. [ 1.304259] RPC: Registered tcp transport module. [ 1.305920] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.308201] NET: Registered protocol family 44 [ 1.309967] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.311969] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.313891] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.316067] PCI: CLS 0 bytes, default 64 [ 1.317744] Unpacking initramfs... [ 2.718796] debug: unmapping init [mem 0xffffa0bb3cc54000-0xffffa0bb3ffbffff] [ 2.722624] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.724812] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.727873] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.271575] Initialise system trusted keyrings [ 3.273289] Key type blacklist registered [ 3.275151] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.284248] zbud: loaded [ 3.287144] *** VALIDATE nfs *** [ 3.288518] *** VALIDATE nfs4 *** [ 3.290118] pstore: using deflate compression [ 3.294367] Platform Keyring initialized [ 3.406091] NET: Registered protocol family 38 [ 3.407650] Key type asymmetric registered [ 3.408902] Asymmetric key parser 'x509' registered [ 3.410282] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.412973] io scheduler mq-deadline registered [ 3.414285] io scheduler kyber registered [ 3.415388] io scheduler bfq registered [ 3.416792] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.419690] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.422804] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.425429] ACPI: Power Button [PWRF] [ 3.430522] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.436437] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.445698] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.450957] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.463039] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.489881] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.516205] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.520225] Non-volatile memory driver v1.3 [ 3.521737] Linux agpgart interface v0.103 [ 3.556197] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.559174] vda: detected capacity change from 0 to 74702848 [ 3.580986] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.583951] vdb: detected capacity change from 0 to 1073741824 [ 3.607834] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.611107] vdc: detected capacity change from 0 to 2621440000 [ 3.626189] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.628301] vdd: detected capacity change from 0 to 2621440000 [ 3.648748] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.650725] vde: detected capacity change from 0 to 4294967296 [ 3.671546] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.673771] vdf: detected capacity change from 0 to 4294967296 [ 3.680503] libphy: Fixed MDIO Bus: probed [ 3.709876] usbcore: registered new interface driver usbserial_generic [ 3.712403] usbserial: USB Serial support registered for generic [ 3.715113] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.720341] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.722503] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.725159] mousedev: PS/2 mouse device common for all mice [ 3.728390] rtc_cmos 00:05: RTC can wake from S4 [ 3.732512] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.734411] rtc_cmos 00:05: registered as rtc0 [ 3.737730] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.740470] intel_pstate: CPU model not supported [ 3.742885] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.747110] hid: raw HID events driver (C) Jiri Kosina [ 3.748921] usbcore: registered new interface driver usbhid [ 3.751031] usbhid: USB HID core driver [ 3.752623] drop_monitor: Initializing network drop monitor service [ 3.755067] Initializing XFRM netlink socket [ 3.757219] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.757455] NET: Registered protocol family 10 [ 3.763115] Segment Routing with IPv6 [ 3.764953] NET: Registered protocol family 17 [ 3.768298] mpls_gso: MPLS GSO support [ 3.775472] RAS: Correctable Errors collector initialized. [ 3.777288] AVX version of gcm_enc/dec engaged. [ 3.778914] AES CTR mode by8 optimization enabled [ 3.873545] sched_clock: Marking stable (3873526344, 0)->(4735430609, -861904265) [ 3.877304] registered taskstats version 1 [ 3.880310] Loading compiled-in X.509 certificates [ 3.882318] zswap: loaded using pool lzo/zbud [ 3.907633] Key type big_key registered [ 3.922499] Key type encrypted registered [ 3.924924] ima: No TPM chip found, activating TPM-bypass! [ 3.927925] ima: Allocated hash algorithm: sha1 [ 3.929793] ima: No architecture policies found [ 3.931472] evm: Initialising EVM extended attributes: [ 3.933425] evm: security.selinux [ 3.934664] evm: security.ima [ 3.935783] evm: security.capability [ 3.937068] evm: HMAC attrs: 0x1 [ 3.939571] rtc_cmos 00:05: setting system clock to 2026-07-30 04:59:15 UTC (1785387555) [ 3.945657] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 3.949119] debug: unmapping init [mem 0xffffffffac782000-0xffffffffaca58fff] [ 3.957086] Write protecting the kernel read-only data: 28672k [ 3.960608] debug: unmapping init [mem 0xffffffffaae03000-0xffffffffaaffffff] [ 3.964016] debug: unmapping init [mem 0xffffffffab714000-0xffffffffab7fffff] [ 3.997272] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.005501] systemd[1]: Detected virtualization kvm. [ 4.007813] systemd[1]: Detected architecture x86-64. [ 4.010274] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.039130] systemd[1]: No hostname configured. [ 4.041584] systemd[1]: Set hostname to . [ 4.044318] random: systemd: uninitialized urandom read (16 bytes read) [ 4.047551] systemd[1]: Initializing machine ID from random generator. [ 4.169538] random: systemd: uninitialized urandom read (16 bytes read) [ 4.171652] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.175398] random: systemd: uninitialized urandom read (16 bytes read) [ 4.177154] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.181792] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.732798] device-mapper: uevent: version 1.0.3 [ 4.734343] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.395151] random: fast init done [ 5.407952] virtio_net virtio0 ens2: renamed from eth0 [ 5.440249] scsi host0: ata_piix [ 5.449766] scsi host1: ata_piix [ 5.451112] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.453182] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.040605] dracut-initqueue[577]: RTNETLINK answers: File exists [ 10.441737] random: crng init done [ 10.442990] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.751480] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.869334] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.171366] SELinux: Disabled at runtime. [ 12.234497] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.243361] systemd[1]: Detected virtualization kvm. [ 12.245088] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.747379] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.749750] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.754784] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.758696] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.761015] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.770365] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.781439] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ 12.909431] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [[ 13.163658] squashfs: version 4.0 (2009/01/31) Phillip Lougher  OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.477584] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.505808] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.630674] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.653792] EDAC sbridge: Ver: 1.1.2 [ 15.619413] Key type dns_resolver registered [ 15.922663] NFS: Registering the id_resolver key type [ 15.925483] Key type id_resolver registered [ 15.927621] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg120-server login: [ 90.012006] hrtimer: interrupt took 6572435 ns [ 124.343487] libcfs: loading out-of-tree module taints kernel. [ 124.411615] Key type ._llcrypt registered [ 124.413386] Key type .llcrypt registered [ 124.560357] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 151.559860] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 153.635802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 153.641914] alg: No test for adler32 (adler32-zlib) [ 155.261632] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 156.399809] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 158.143185] Key type lgssc registered [ 160.552098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 180.406318] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 234.763477] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 251.092915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 251.165072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 252.651129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 252.698645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 252.824161] Lustre: lustre-MDT0000: new disk, initializing [ 252.908410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.922421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 257.801415] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.930341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 274.053325] Lustre: 6534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 274.095557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 274.101681] Lustre: Skipped 1 previous similar message [ 274.223489] Lustre: lustre-MDT0001: new disk, initializing [ 274.369073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 274.438750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 274.451709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 280.631120] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 286.196819] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 297.958951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 298.185021] Lustre: lustre-OST0000: new disk, initializing [ 298.187656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 298.190850] Lustre: 8471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 298.248743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 305.708282] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 305.742153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 305.756560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 305.816171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 326.676364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 326.913537] Lustre: lustre-OST0001: new disk, initializing [ 326.917796] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 326.925129] Lustre: 9544:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 327.014869] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 332.413482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 332.429560] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 332.512566] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 335.428652] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 350.026807] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 357.459808] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 365.268288] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing check_logdir /tmp/testlogs/ [ 370.939135] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing yml_node [ 378.695636] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 381.856458] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 384.550539] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 386.398091] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 30 01:05:36 EDT 2026 [ 406.878823] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 409.255607] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 428.005624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 428.013612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.039059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 430.058417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.059628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 430.102041] Lustre: Skipped 1 previous similar message [ 430.125395] Lustre: Skipped 3 previous similar messages [ 435.174079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.291169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.300822] Lustre: Skipped 6 previous similar messages [ 441.311150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 441.600565] Lustre: server umount lustre-MDT0000 complete [ 446.718395] LustreError: 6524:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785387998 with bad export cookie 13446653986260962140 [ 446.721209] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 446.726549] LustreError: 6524:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 450.528077] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.529352] LustreError: 6541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.551767] Lustre: Skipped 1 previous similar message [ 450.588494] LustreError: 6541:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 450.644821] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 450.648090] Lustre: Skipped 1 previous similar message [ 451.552319] LustreError: 9756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.576060] LustreError: 9756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 456.681989] LustreError: 6539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.719717] LustreError: 6539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 461.279121] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 461.687395] Lustre: server umount lustre-MDT0001 complete [ 467.120393] Lustre: server umount lustre-OST0000 complete [ 471.995038] Lustre: server umount lustre-OST0001 complete [ 481.148470] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 493.274370] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 548.240214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.586772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 548.626889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 548.720764] Lustre: lustre-MDT0000: new disk, initializing [ 548.797570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 548.815572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 555.321606] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 568.854726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.064918] Lustre: 15108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 569.173307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 569.178388] Lustre: Skipped 1 previous similar message [ 569.335555] Lustre: lustre-MDT0001: new disk, initializing [ 569.541231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 569.594348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 569.611364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 575.263436] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 586.653545] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 594.494286] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 601.873201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 602.387056] Lustre: lustre-OST0000: new disk, initializing [ 602.395786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 602.410353] Lustre: 17080:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 602.578542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 604.266093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 604.278192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 604.396517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 609.911680] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 622.497683] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 628.156746] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 628.467022] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 635.216224] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 635.606205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 640.480937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 640.483453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 640.489214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 640.511052] Lustre: Skipped 2 previous similar messages [ 640.534194] Lustre: Skipped 5 previous similar messages [ 644.981173] Lustre: server umount lustre-OST0000 complete [ 651.232640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 651.245265] LustreError: Skipped 1 previous similar message [ 651.247847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.394317] Lustre: server umount lustre-MDT0000 complete [ 660.961094] LustreError: 15116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 660.993346] LustreError: 15116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 662.799180] LustreError: 15101:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785388214 with bad export cookie 13446653986260963673 [ 662.807746] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 662.820597] LustreError: 15101:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 663.316593] Lustre: server umount lustre-MDT0001 complete [ 672.040040] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:10:22 (1785388222) [ 693.899444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.288188] LustreError: 19958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 694.343614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 700.176523] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 703.482240] LustreError: 19959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 703.519585] LustreError: 19959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 709.738665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.899950] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 724.522197] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 734.508744] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 740.158362] Lustre: Failing over lustre-MDT0000 [ 740.844450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 740.852811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 740.870980] Lustre: Skipped 1 previous similar message [ 740.899931] LustreError: 21055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 740.919289] LustreError: 21055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 741.262617] Lustre: server umount lustre-MDT0000 complete [ 760.331203] Lustre: 3664:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785388295/real 1785388295] req@ffffa0bbb735dc00 x1872114700430848/t0(0) o400->MGC192.168.201.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785388311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 760.464145] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 777.185370] LustreError: 19959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 777.200451] LustreError: 19959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 782.920973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 786.464176] LustreError: 3662:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0bbb735ca80 x1872114700434944/t0(0) o250->MGC192.168.201.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 787.213894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 787.222082] Lustre: Skipped 1 previous similar message [ 787.332636] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 792.567296] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 792.619919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 792.684418] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 799.114397] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 820.339829] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 827.877538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.922390] Lustre: Skipped 1 previous similar message [ 827.937990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 827.953013] Lustre: Skipped 4 previous similar messages [ 835.881628] Lustre: server umount lustre-MDT0000 complete [ 844.083369] LustreError: 20694:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785388395 with bad export cookie 13446653986260965087 [ 844.106537] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 844.123331] LustreError: 20694:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 845.647893] Lustre: server umount lustre-MDT0001 complete [ 858.992671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.086640] LustreError: 23306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 860.102584] LustreError: 23306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 868.232971] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 878.857709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.874690] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 895.782931] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 904.169826] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 906.973121] Lustre: Failing over lustre-MDT0000 [ 907.260513] Lustre: server umount lustre-MDT0000 complete [ 910.307424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 910.308250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.322512] Lustre: Skipped 2 previous similar messages [ 925.665586] Lustre: 3664:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785388461/real 1785388461] req@ffffa0bbbf971500 x1872114700468992/t0(0) o400->MGC192.168.201.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785388477 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 925.686712] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 927.289622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 935.907443] LustreError: 3662:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0ba836fd880 x1872114700471808/t0(0) o250->MGC192.168.201.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 936.379492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 936.385513] Lustre: Skipped 2 previous similar messages [ 936.466779] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 941.544930] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 941.569502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 941.576677] Lustre: Skipped 1 previous similar message [ 941.617355] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 941.765703] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 956.170877] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 962.019996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 962.031458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 962.033482] Lustre: Skipped 2 previous similar messages [ 968.055723] Lustre: server umount lustre-MDT0000 complete [ 972.375497] LustreError: 24043:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785388523 with bad export cookie 13446653986260966025 [ 972.380255] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 972.386972] LustreError: 24043:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 972.869316] Lustre: server umount lustre-MDT0001 complete [ 984.555879] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:15:34 (1785388534) [ 1006.350154] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 1017.074703] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1079.050562] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1097.142891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.755765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1097.847352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1098.031539] Lustre: lustre-MDT0000: new disk, initializing [ 1098.222075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1104.246128] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1117.496980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.719496] Lustre: 29706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1118.037649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1124.484679] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1137.599699] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1146.665783] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1157.387033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1157.899676] Lustre: 31675:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1159.315034] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1159.411185] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1159.496312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1159.516144] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1166.983638] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1179.474462] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1187.693502] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1221.095838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1221.099046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1221.141336] Lustre: Skipped 2 previous similar messages [ 1221.155803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1221.159616] Lustre: Skipped 3 previous similar messages [ 1227.156881] Lustre: server umount lustre-OST0000 complete [ 1240.988893] Lustre: server umount lustre-MDT0000 complete [ 1241.569892] LustreError: 29717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.611388] LustreError: 29717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 1247.175974] LustreError: 29697:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785388798 with bad export cookie 13446653986260966431 [ 1247.177383] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1247.185027] LustreError: 29697:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1247.833460] Lustre: server umount lustre-MDT0001 complete [ 1261.221661] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1266.440079] Key type lgssc unregistered [ 1267.086644] LNet: 33656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.119062] LNetError: 33656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.157531] LNet: Removed LNI 192.168.201.120@tcp [ 1269.904267] Key type .llcrypt unregistered [ 1269.906853] Key type ._llcrypt unregistered [ 1303.487258] Key type ._llcrypt registered [ 1303.488816] Key type .llcrypt registered [ 1303.697829] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1305.728014] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1305.860086] alg: No test for adler32 (adler32-zlib) [ 1307.174506] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 1307.512572] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1309.175502] Key type lgssc registered [ 1310.818942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1321.537482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1321.559339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.453456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1329.397573] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1332.713385] LustreError: 35071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.744579] LustreError: 35071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1337.838745] LustreError: 35072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.709730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.111470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1346.755398] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1357.077239] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1369.065637] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1382.336897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1382.707398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.804795] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1390.603134] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1393.087504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1401.600279] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1423.109546] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:22:52 (1785388972) [ 1426.345148] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1429.488558] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1441.518923] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:23:10 (1785388990) [ 1454.059896] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:23:24 (1785389004) [ 1455.539234] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1466.783776] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1466.786727] Lustre: Skipped 2 previous similar messages [ 1469.130034] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:23:39 (1785389019) [ 1471.650610] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1471.655801] Lustre: Skipped 2 previous similar messages [ 1482.001311] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1482.008434] Lustre: Skipped 1 previous similar message [ 1511.328931] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1511.334819] Lustre: Skipped 5 previous similar messages [ 1515.581856] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:24:25 (1785389065) [ 1520.540526] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1527.364747] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1540.807875] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:24:51 (1785389091) [ 1543.401687] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1543.407773] Lustre: Skipped 3 previous similar messages [ 1561.223750] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:25:11 (1785389111) [ 1580.512094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1580.517942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1580.520717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1580.531819] Lustre: Skipped 1 previous similar message [ 1585.632452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1585.644895] Lustre: Skipped 2 previous similar messages [ 1590.752835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1594.335253] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1594.812651] Lustre: server umount lustre-OST0000 complete [ 1601.204252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1601.212901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1601.228421] Lustre: Skipped 1 previous similar message [ 1601.259490] Lustre: Skipped 2 previous similar messages [ 1606.629778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1606.644180] Lustre: Skipped 1 previous similar message [ 1607.938645] Lustre: server umount lustre-MDT0000 complete [ 1611.745167] LustreError: 35071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1611.772457] LustreError: 35071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1613.156991] LustreError: 38641:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785389164 with bad export cookie 6946487183941435025 [ 1613.169842] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1613.174267] LustreError: 38641:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1614.090093] Lustre: server umount lustre-MDT0001 complete [ 1630.587783] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 1634.400161] Key type lgssc unregistered [ 1634.855812] LNet: 42944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.860718] LNetError: 42944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.870882] LNet: Removed LNI 192.168.201.120@tcp [ 1635.944343] Key type .llcrypt unregistered [ 1635.948272] Key type ._llcrypt unregistered [ 1661.419338] Key type ._llcrypt registered [ 1661.421069] Key type .llcrypt registered [ 1661.581122] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 1684.036500] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 1686.729835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1686.855461] alg: No test for adler32 (adler32-zlib) [ 1688.432097] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 1689.283378] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 1691.271277] Key type lgssc registered [ 1693.621851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1749.618867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1749.654302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.253147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1751.349183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1751.511533] Lustre: lustre-MDT0000: new disk, initializing [ 1751.677104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1751.693038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1757.314842] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1771.152227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.410239] Lustre: 46106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1771.581491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1771.598268] Lustre: Skipped 1 previous similar message [ 1771.745453] Lustre: lustre-MDT0001: new disk, initializing [ 1771.879446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1771.986059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1772.000581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1778.019449] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1791.920840] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1802.395727] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1811.877281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1812.250493] Lustre: lustre-OST0000: new disk, initializing [ 1812.261397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1812.270479] Lustre: 48077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1812.442017] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1820.264174] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1820.273908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1820.304271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1820.457172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1831.440259] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1838.004206] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1838.286510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1844.081444] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1844.545352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1851.364468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.378727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1851.378789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1851.398840] LustreError: Skipped 1 previous similar message [ 1853.200325] Lustre: server umount lustre-OST0000 complete [ 1859.056062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1859.070729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.091151] Lustre: Skipped 1 previous similar message [ 1859.098385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1859.114128] Lustre: Skipped 1 previous similar message [ 1861.602087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1861.607524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.728154] Lustre: server umount lustre-MDT0000 complete [ 1868.086673] LustreError: 46099:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785389419 with bad export cookie 17484875060655758582 [ 1868.089088] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1868.100734] LustreError: 46099:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1868.431714] Lustre: server umount lustre-MDT0001 complete [ 1880.554394] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:30:30 (1785389430) [ 1888.839680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1889.658634] LustreError: 50537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.673608] LustreError: 50537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1889.798523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.881852] LustreError: 50538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1895.387539] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1900.002336] LustreError: 50537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1905.126176] LustreError: 50538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1906.411829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.368204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1918.675021] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1933.447994] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1941.106287] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1950.109203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1950.604240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.785968] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 1970.957991] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1976.742635] LustreError: 53070:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1978.033926] Lustre: Modifying parameter general.timeout=1 in log params [ 1979.457448] Lustre: Modifying parameter general.timeout=2 in log params [ 1981.651795] Lustre: Modifying parameter general.timeout=4 in log params [ 1981.660159] Lustre: Skipped 1 previous similar message [ 1986.765830] Lustre: Modifying parameter general.timeout=8 in log params [ 1986.768008] Lustre: Skipped 3 previous similar messages [ 1994.867512] Lustre: Modifying parameter general.timeout=14 in log params [ 1994.875322] Lustre: Skipped 5 previous similar messages [ 2011.610879] Lustre: Modifying parameter general.timeout=30 in log params [ 2011.613953] Lustre: Skipped 15 previous similar messages [ 2043.858258] Lustre: Modifying parameter general.timeout=59 in log params [ 2043.861605] Lustre: Skipped 28 previous similar messages [ 2109.070066] Lustre: Modifying parameter general.timeout=112 in log params [ 2109.076760] Lustre: Skipped 52 previous similar messages [ 2148.566430] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:34:58 (1785389698) [ 2152.827138] Lustre: 60038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 2154.948990] Lustre: 60136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 2157.507550] Lustre: 60251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 2162.489508] Lustre: 60447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 2162.504417] Lustre: 60447:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2196.240958] Lustre: 61921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 2196.247327] Lustre: 61921:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 2196.277988] LustreError: 61921:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 2202.523469] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:35:52 (1785389752) [ 2224.612145] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2224.614727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2224.631775] Lustre: Skipped 1 previous similar message [ 2224.643713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2224.658585] Lustre: Skipped 3 previous similar messages [ 2226.519645] Lustre: server umount lustre-MDT0000 complete [ 2229.731090] LustreError: 50537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.760983] LustreError: 50537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2230.725793] LustreError: 50519:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785389782 with bad export cookie 17484875060655759464 [ 2230.728385] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2230.747430] LustreError: 50519:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2231.001704] Lustre: server umount lustre-MDT0001 complete [ 2246.210811] Lustre: server umount lustre-OST0000 complete [ 2280.263811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.487628] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2280.534125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2280.546485] Lustre: Skipped 40 previous similar messages [ 2280.853450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2284.837618] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2296.568333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2296.702842] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2296.750485] Lustre: 64455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2302.771740] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2311.692149] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2318.140692] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2324.697840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2324.911802] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2325.104678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2325.111709] Lustre: Skipped 1 previous similar message [ 2331.976833] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2340.200845] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2341.696147] Lustre: 67024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2353.639125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2353.644422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2353.657434] Lustre: Skipped 1 previous similar message [ 2353.672043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2355.678444] Lustre: server umount lustre-OST0000 complete [ 2363.360543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2363.374838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.387935] Lustre: Skipped 1 previous similar message [ 2363.397520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2363.406666] Lustre: Skipped 2 previous similar messages [ 2366.848769] Lustre: server umount lustre-MDT0000 complete [ 2369.509737] LustreError: 64465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.527414] LustreError: 64465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2373.304679] LustreError: 64449:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785389924 with bad export cookie 17484875060655762873 [ 2373.316136] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2373.325917] LustreError: 64449:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2373.861118] Lustre: server umount lustre-MDT0001 complete [ 2393.505233] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 2397.023707] Key type lgssc unregistered [ 2397.373418] LNet: 68262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2397.383682] LNetError: 68262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.401781] LNet: Removed LNI 192.168.201.120@tcp [ 2398.290836] Key type .llcrypt unregistered [ 2398.298432] Key type ._llcrypt unregistered [ 2430.708382] Key type ._llcrypt registered [ 2430.710323] Key type .llcrypt registered [ 2430.831295] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2432.033894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2432.090509] alg: No test for adler32 (adler32-zlib) [ 2433.417252] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 2433.662552] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 2435.327711] Key type lgssc registered [ 2436.727917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2446.407798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2446.429292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2448.097645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.131404] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2457.585641] LustreError: 69677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.618434] LustreError: 69677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2462.690499] LustreError: 69676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2467.823301] LustreError: 69677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.621627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2470.769864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2476.891567] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2486.393290] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2493.223190] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2501.681602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2502.268600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2510.087795] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2519.100751] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2527.563425] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2538.465878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.469865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2538.484746] Lustre: Skipped 2 previous similar messages [ 2538.496249] Lustre: Skipped 2 previous similar messages [ 2543.585610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2544.485258] Lustre: server umount lustre-MDT0000 complete [ 2548.760285] LustreError: 69673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.784449] LustreError: 69673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2549.809859] LustreError: 69656:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785390101 with bad export cookie 8608913034662753654 [ 2549.819734] LustreError: 69656:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2549.819837] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2550.357184] Lustre: server umount lustre-MDT0001 complete [ 2566.608803] Lustre: server umount lustre-OST0000 complete [ 2605.448647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.898865] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2605.943517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2606.207117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2612.272900] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2623.136433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.210318] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2623.232438] Lustre: 74565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2623.775755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2630.993327] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2644.303423] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2651.975257] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2659.571963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.764156] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2660.111404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.700495] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2676.696307] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2678.610752] Lustre: 77142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2680.881717] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2688.482337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2688.492543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.508387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2688.512660] Lustre: Skipped 2 previous similar messages [ 2688.992523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2692.926501] Lustre: server umount lustre-OST0000 complete [ 2699.234189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.236105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.256456] Lustre: Skipped 2 previous similar messages [ 2699.267114] Lustre: Skipped 1 previous similar message [ 2704.353698] LustreError: 74576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.382435] LustreError: 74576:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2706.047497] Lustre: server umount lustre-MDT0000 complete [ 2710.391351] LustreError: 74556:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785390261 with bad export cookie 8608913034662754501 [ 2710.392023] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2710.404778] LustreError: 74556:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2710.927760] Lustre: server umount lustre-MDT0001 complete [ 2724.978984] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 2728.341415] Key type lgssc unregistered [ 2728.594673] LNet: 78355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2728.602722] LNetError: 78355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2729.640240] LNet: Removed LNI 192.168.201.120@tcp [ 2730.708154] Key type .llcrypt unregistered [ 2730.710072] Key type ._llcrypt unregistered [ 2760.662109] Key type ._llcrypt registered [ 2760.665733] Key type .llcrypt registered [ 2760.811978] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 2762.027849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2762.147829] alg: No test for adler32 (adler32-zlib) [ 2763.242390] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 2763.497619] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 2765.143157] Key type lgssc registered [ 2766.479178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2774.268199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2774.289444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.811913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2781.736597] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2786.291847] LustreError: 79770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.311960] LustreError: 79770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2791.172742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.393100] LustreError: 79771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.813958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2796.696809] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2804.660688] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2810.999388] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2818.061070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2818.682708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2826.153678] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 2835.148288] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2843.035926] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2844.229860] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2846.015376] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2846.018870] Lustre: Skipped 1 previous similar message [ 2848.128662] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2848.134880] Lustre: Skipped 1 previous similar message [ 2852.607534] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2852.612757] Lustre: Skipped 4 previous similar messages [ 2861.044476] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2861.053189] Lustre: Skipped 8 previous similar messages [ 2878.325852] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2878.331031] Lustre: Skipped 15 previous similar messages [ 2910.650531] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2910.659788] Lustre: Skipped 26 previous similar messages [ 2958.644930] Lustre: 87883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2962.494774] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:48:32 (1785390512) [ 2965.056322] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2967.659635] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:48:37 (1785390517) [ 2974.696792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2974.715853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.740840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2976.745925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2976.761757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2976.764754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2976.780252] Lustre: Skipped 1 previous similar message [ 2979.555704] Lustre: server umount lustre-OST0000 complete [ 2986.466807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2986.476054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2986.501702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.648637] Lustre: server umount lustre-MDT0000 complete [ 2992.099682] LustreError: 79770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.131355] LustreError: 79770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2997.221183] LustreError: 80778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.243259] LustreError: 80778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2997.315503] LustreError: 79752:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785390548 with bad export cookie 16817031445532283153 [ 2997.328130] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2998.281636] Lustre: server umount lustre-MDT0001 complete [ 3016.965873] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 3020.442681] Key type lgssc unregistered [ 3020.762732] LNet: 89152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3020.768589] LNetError: 89152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3020.777911] LNet: Removed LNI 192.168.201.120@tcp [ 3021.469199] Key type .llcrypt unregistered [ 3021.472764] Key type ._llcrypt unregistered [ 3026.372535] Key type ._llcrypt registered [ 3026.374492] Key type .llcrypt registered [ 3026.473396] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 3030.703393] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules [ 3031.647586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3031.709066] alg: No test for adler32 (adler32-zlib) [ 3032.734590] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 3032.914879] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 3034.557860] Key type lgssc registered [ 3036.063614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3037.830242] LustreError: 90355:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 3060.752568] LustreError: 90355:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 3065.456123] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3074.712562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3074.723818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.532050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.965844] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3085.812407] LustreError: 91388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.843536] LustreError: 91388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3090.605753] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:50:41 (1785390641) [ 3090.914334] LustreError: 91387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3095.489660] LustreError: 91419:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3095.503858] LustreError: 91419:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3095.512205] LustreError: 91419:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 3095.770628] Lustre: server umount lustre-MDT0000 complete [ 3112.625513] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 3114.955387] Key type lgssc unregistered [ 3115.274707] LNet: 92913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3115.280578] LNetError: 92913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3116.327797] LNet: Removed LNI 192.168.201.120@tcp [ 3117.966824] Key type .llcrypt unregistered [ 3117.978075] Key type ._llcrypt unregistered [ 3145.844528] Key type ._llcrypt registered [ 3145.845984] Key type .llcrypt registered [ 3146.048692] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3148.031854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3148.234195] alg: No test for adler32 (adler32-zlib) [ 3149.571270] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 3149.924297] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 3151.631168] Key type lgssc registered [ 3153.134656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3161.377437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3161.430314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.217426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3169.937365] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3173.343887] LustreError: 94136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.374639] LustreError: 94136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3178.465146] LustreError: 94135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3180.280194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.859834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3186.849905] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3197.100431] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3206.914891] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3215.596099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3216.524136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.034357] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3235.540791] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3289.281532] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:53:59 (1785390839) [ 3290.897957] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 3293.292535] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:54:03 (1785390843) [ 3298.277306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3298.291348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.323201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.811868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.812663] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.817787] Lustre: Skipped 1 previous similar message [ 3299.853771] Lustre: Skipped 1 previous similar message [ 3303.995391] Lustre: server umount lustre-MDT0000 complete [ 3304.962431] LustreError: 94130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.993847] LustreError: 94130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3308.549483] LustreError: 94116:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785390860 with bad export cookie 14964172773323441449 [ 3308.570601] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3308.575060] LustreError: 94116:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3309.313204] Lustre: server umount lustre-MDT0001 complete [ 3325.025984] Lustre: server umount lustre-OST0000 complete [ 3337.915032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3338.364602] LustreError: 98425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3338.538193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3343.640595] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3343.861265] LustreError: 98426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.062319] LustreError: 98425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.075217] LustreError: 98425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3353.546778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.368066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3360.103986] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3370.788824] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3379.686667] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3396.209307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3396.392102] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3396.398719] LustreError: 98414:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3396.419249] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3396.432601] LustreError: 100513:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3396.435790] LustreError: 100513:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3396.456928] LustreError: 100513:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3396.592452] Lustre: server umount lustre-OST0000 complete [ 3396.599410] LustreError: 100513:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3405.834805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3406.020960] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3406.026160] LustreError: 98412:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3406.045510] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3406.053474] LustreError: 100979:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3406.060122] LustreError: 100979:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3406.063574] LustreError: 100979:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3406.130690] Lustre: server umount lustre-OST0000 complete [ 3406.136372] LustreError: 100979:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3417.447709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3417.570631] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3417.573763] Lustre: Found index 0 for lustre-OST0000, updating log [ 3417.578179] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3417.634677] Lustre: lustre-OST0000: new disk, initializing [ 3417.637749] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3417.644188] Lustre: 101514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3417.863497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3425.197937] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3430.012645] Lustre: Failing over lustre-OST0000 [ 3430.131802] Lustre: server umount lustre-OST0000 complete [ 3436.004957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3436.020628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3436.059968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3436.081719] Lustre: Skipped 1 previous similar message [ 3438.564903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.573603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3438.598109] Lustre: Skipped 1 previous similar message [ 3443.683160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3448.800827] Lustre: 93447:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785390984/real 1785390984] req@00000000fc7935c4 x1872117839798912/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785391000 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3448.831304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.311228] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3450.047623] Lustre: server umount lustre-MDT0000 complete [ 3450.851488] Lustre: 93444:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785390986/real 1785390986] req@00000000876b62ff x1872117839799168/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785391002 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3453.921424] LustreError: 98420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3453.944713] LustreError: 98420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3454.880171] Lustre: 93445:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785390990/real 1785390990] req@000000005f3c1617 x1872117839800704/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785391006 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3455.255190] LustreError: 98407:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785391006 with bad export cookie 14964172773323444193 [ 3455.276895] LustreError: 98407:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3455.278398] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3455.893347] Lustre: server umount lustre-MDT0001 complete [ 3466.166840] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:56:56 (1785391016) [ 3487.294408] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3499.864523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.342825] LustreError: 103867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3500.439768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3505.621554] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3515.471722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3521.822306] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3525.913819] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3534.220803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3536.153218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.173615] Lustre: Skipped 1 previous similar message [ 3536.189611] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3541.243562] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3541.295581] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3541.296115] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3544.843404] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3554.533770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3554.666886] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3554.801467] Lustre: lustre-OST0001: new disk, initializing [ 3554.815445] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3554.828628] Lustre: 106186:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3561.547627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3561.558285] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3561.752369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3563.301933] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3570.812866] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3582.839535] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3587.565186] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.567184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3587.576807] Lustre: Skipped 2 previous similar messages [ 3587.589788] Lustre: Skipped 5 previous similar messages [ 3593.244216] Lustre: server umount lustre-MDT0001 complete [ 3596.445354] LustreError: 103863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3596.458141] LustreError: 103863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3608.347466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.509028] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3608.510955] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3608.587274] Lustre: 103856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3609.528898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3609.540774] Lustre: Skipped 1 previous similar message [ 3610.202647] LustreError: 107729:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3610.209805] LustreError: 107729:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3610.222834] LustreError: 103859:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3610.372644] LustreError: 107741:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.120@tcp: cfg command failed: rc = -17 [ 3610.377682] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3610.377682] [ 3614.715183] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3617.659085] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3630.124393] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:59:40 (1785391180) [ 3657.340400] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3679.964773] Lustre: 109824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3708.333314] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3713.378482] Lustre: 110972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3808.755517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3808.773432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3808.789661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3808.805536] Lustre: Skipped 4 previous similar messages [ 3812.971044] Lustre: server umount lustre-MDT0000 complete [ 3814.374905] LustreError: 107170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3814.386534] LustreError: 107170:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 3816.780302] LustreError: 103849:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785391368 with bad export cookie 14964172773323445194 [ 3816.783282] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3816.795098] LustreError: 103849:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3817.404230] Lustre: server umount lustre-MDT0001 complete [ 3832.607410] Lustre: server umount lustre-OST0000 complete [ 3835.679109] Lustre: 93446:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785391371/real 1785391371] req@ffffa0bbba702680 x1872117840077184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785391387 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3838.294586] Lustre: server umount lustre-OST0001 complete [ 3845.163236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3857.225927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3868.098190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3880.189119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3906.389333] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 3920.128332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3920.160435] Lustre: lustre-MDT0000: reset Object Index mappings [ 3920.854421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3927.624916] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3940.998110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3941.034596] Lustre: lustre-MDT0001: reset Object Index mappings [ 3941.348148] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3941.360257] Lustre: Skipped 4 previous similar messages [ 3949.949256] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3954.530703] Lustre: 116075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3965.048702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3966.786836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3966.810618] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3972.456524] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 3982.859314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3988.987021] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3988.988298] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3991.914711] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4002.010552] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4006.485915] Lustre: 117956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4077.309584] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 02:07:07 (1785391627) [ 4105.184412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4105.201683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4105.224139] Lustre: Skipped 4 previous similar messages [ 4105.243606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4109.963302] Lustre: server umount lustre-MDT0000 complete [ 4111.843305] LustreError: 114921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.870613] LustreError: 114921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 4114.806898] LustreError: 114908:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785391666 with bad export cookie 14964172773323492038 [ 4114.832590] LustreError: 114908:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4114.857295] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4129.247619] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4129.875688] Lustre: server umount lustre-MDT0001 complete [ 4140.906477] Lustre: server umount lustre-OST0000 complete [ 4151.357533] Lustre: server umount lustre-OST0001 complete [ 4158.797943] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 4171.221882] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4221.016212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.346973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4221.375795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4221.442452] Lustre: lustre-MDT0000: new disk, initializing [ 4221.519624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4221.523435] Lustre: Skipped 3 previous similar messages [ 4221.534063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4228.979862] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4238.738957] Lustre: server umount lustre-MDT0000 complete [ 4253.257821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4253.669914] Lustre: Found index 0 for lustre-MDT0000, updating log [ 4253.695953] Lustre: 122239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4253.723393] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 4264.261249] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4275.293515] Lustre: server umount lustre-MDT0000 complete [ 4294.719678] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:10:44 (1785391844) [ 4296.985640] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 4299.696924] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:10:49 (1785391849) [ 4322.545069] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 4338.471935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.928271] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4361.028836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4361.232842] Lustre: 124293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4361.329465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4361.335493] Lustre: Skipped 1 previous similar message [ 4361.484616] Lustre: lustre-MDT0001: new disk, initializing [ 4361.733364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4361.753539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4367.926931] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4374.588409] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4383.255858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4383.467400] Lustre: lustre-OST0000: new disk, initializing [ 4383.470609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4383.475578] Lustre: 125870:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4384.738987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4384.755660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4384.981177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4392.198514] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4405.572618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4405.708684] Lustre: lustre-OST0001: new disk, initializing [ 4405.710969] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4405.714773] Lustre: 126741:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4406.968262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4406.982101] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4407.274287] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4415.721612] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 4426.900638] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4431.521940] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4937.869448] Lustre: 133178:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4937.884684] Lustre: 133178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4937.895503] Lustre: 133178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4937.905069] Lustre: 133178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4937.916612] Lustre: 133178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4937.927126] Lustre: 133178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 5310.603237] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:27:39 (1785392859) [ 5328.865254] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5328.873929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5328.882069] Lustre: Skipped 8 previous similar messages [ 5328.889605] Lustre: Skipped 12 previous similar messages [ 5335.716776] Lustre: server umount lustre-MDT0000 complete [ 5339.121229] LustreError: 127575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5339.156544] LustreError: 127575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5342.861245] LustreError: 124285:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785392894 with bad export cookie 14964172773323564005 [ 5342.882443] LustreError: 124285:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5342.887517] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5344.226759] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5344.250598] Lustre: Skipped 1 previous similar message [ 5347.297170] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5347.315389] Lustre: Skipped 8 previous similar messages [ 5350.150740] Lustre: server umount lustre-MDT0001 complete [ 5370.335269] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5370.511884] Lustre: server umount lustre-OST0000 complete [ 5384.119880] Lustre: server umount lustre-OST0001 complete [ 5395.100211] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 5412.630496] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 5469.715726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5470.167651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5470.260382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5470.415830] Lustre: lustre-MDT0000: new disk, initializing [ 5470.558616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5470.576131] Lustre: Skipped 5 previous similar messages [ 5470.644368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5476.605255] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5495.652359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5496.090709] Lustre: 136277:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5496.213314] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5496.219336] Lustre: Skipped 1 previous similar message [ 5496.355417] Lustre: lustre-MDT0001: new disk, initializing [ 5496.658178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5496.693674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5504.707743] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5517.690972] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5525.644920] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5536.617224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5537.331743] Lustre: lustre-OST0000: new disk, initializing [ 5537.364699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5537.383850] Lustre: 138248:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5537.676523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5537.687889] Lustre: Skipped 1 previous similar message [ 5539.201237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5539.224274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5539.496674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5547.695730] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 5562.095665] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5576.836700] Lustre: lustre-MDD0000: changelog on [ 5581.484874] Lustre: lustre-MDD0001: changelog on [ 7896.298804] Lustre: lustre-MDD0001: changelog off [ 7898.813742] Lustre: 138129:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 7900.034268] Lustre: lustre-MDD0000: changelog off [ 7909.859384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7909.873931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7909.882619] Lustre: Skipped 1 previous similar message [ 7909.893257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7909.910025] Lustre: Skipped 1 previous similar message [ 7913.588939] Lustre: server umount lustre-OST0000 complete [ 7918.563424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7918.584631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7918.627452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7924.136178] Lustre: server umount lustre-MDT0000 complete [ 7925.221162] LustreError: 138129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7925.236694] LustreError: 138129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7928.245203] LustreError: 136268:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785395479 with bad export cookie 14964172773324391755 [ 7928.266109] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7928.583810] Lustre: server umount lustre-MDT0001 complete [ 7948.267192] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 7955.513845] Key type lgssc unregistered [ 7956.259492] LNet: 147483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7956.269153] LNetError: 147483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7956.299455] LNet: Removed LNI 192.168.201.120@tcp [ 7958.956638] Key type .llcrypt unregistered [ 7958.967866] Key type ._llcrypt unregistered [ 7989.161119] Key type ._llcrypt registered [ 7989.165206] Key type .llcrypt registered [ 7989.279561] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 8006.627293] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8008.592622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8008.711509] alg: No test for adler32 (adler32-zlib) [ 8009.794258] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 8010.160152] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 8011.831215] Key type lgssc registered [ 8013.307772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8066.346974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8066.408503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8067.991936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8068.070983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8068.268658] Lustre: lustre-MDT0000: new disk, initializing [ 8068.407602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8068.427700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8075.129832] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8087.110473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8087.253711] Lustre: 150691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8087.307535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8087.312551] Lustre: Skipped 1 previous similar message [ 8087.406916] Lustre: lustre-MDT0001: new disk, initializing [ 8087.555584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8087.611032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8087.622589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8093.281398] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8104.713992] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8112.038218] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8120.809577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8121.083807] Lustre: lustre-OST0000: new disk, initializing [ 8121.088802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8121.095637] Lustre: 152661:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8121.196649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8129.209179] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8130.624253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8130.634201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8130.795956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8139.320893] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8146.415533] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_module obdecho/obdecho [ 8148.314671] LustreError: 153664:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 8148.325133] LustreError: 153664:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 8150.542065] LustreError: 153761:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 8166.886302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8166.890871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8166.902419] Lustre: Skipped 1 previous similar message [ 8166.915172] Lustre: Skipped 2 previous similar messages [ 8171.700425] Lustre: server umount lustre-MDT0000 complete [ 8176.437377] LustreError: 152660:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785395727 with bad export cookie 3368839045200281502 [ 8176.453587] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8176.456128] LustreError: 152660:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8176.692957] Lustre: server umount lustre-MDT0001 complete [ 8192.282390] Lustre: server umount lustre-OST0000 complete [ 8220.090971] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 8232.691369] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8289.679275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.892580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8289.915908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8289.988975] Lustre: lustre-MDT0000: new disk, initializing [ 8290.064510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8290.093423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8295.733392] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8307.674689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8307.854726] Lustre: 157483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8307.924137] LustreError: 157493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8307.950721] LustreError: 157493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 8307.963467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8307.969123] Lustre: Skipped 1 previous similar message [ 8308.081708] Lustre: lustre-MDT0001: new disk, initializing [ 8308.203551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8308.256407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8308.282649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8313.970793] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8324.953941] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8332.773328] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8342.720760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8343.162560] Lustre: lustre-OST0000: new disk, initializing [ 8343.167778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8343.172059] Lustre: 159450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8343.346691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8349.799738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8349.815226] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8349.986047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8352.850217] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8364.925756] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8373.572492] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8373.968178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8381.183320] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8381.527918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8386.020327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8386.022756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8386.028708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8386.051426] Lustre: Skipped 2 previous similar messages [ 8391.151372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8391.160265] Lustre: Skipped 1 previous similar message [ 8396.259001] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8399.327150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8399.962639] Lustre: server umount lustre-OST0000 complete [ 8405.483927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8405.488941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8405.497065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8405.500864] Lustre: Skipped 1 previous similar message [ 8411.626361] LustreError: 157490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8411.643188] LustreError: 157490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8411.795392] Lustre: server umount lustre-MDT0000 complete [ 8416.738890] LustreError: 158851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8416.762224] LustreError: 158851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8417.865779] LustreError: 157473:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785395969 with bad export cookie 3368839045200282461 [ 8417.881517] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8417.888545] LustreError: 157473:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8418.821196] Lustre: server umount lustre-MDT0001 complete [ 8425.826078] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 03:19:35 (1785395975) [ 8427.906939] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 8430.073634] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 03:19:40 (1785395980) [ 8437.963392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8438.387685] LustreError: 161867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8438.460534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8443.895583] LustreError: 161868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8444.136713] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8453.103191] LustreError: 161867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8453.135515] LustreError: 161867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8454.978898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8461.784846] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8472.604361] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8480.078948] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8489.192487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8489.518882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8489.528488] Lustre: Skipped 1 previous similar message [ 8497.328547] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8507.899712] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8516.068898] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8516.079506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8516.090804] Lustre: Skipped 2 previous similar messages [ 8516.112431] Lustre: Skipped 4 previous similar messages [ 8520.583783] Lustre: server umount lustre-MDT0000 complete [ 8521.185755] LustreError: 164361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8521.210737] LustreError: 164361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8524.686336] LustreError: 161848:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785396076 with bad export cookie 3368839045200283343 [ 8524.688066] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8524.700340] LustreError: 161848:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8526.312797] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8526.331388] Lustre: Skipped 1 previous similar message [ 8531.497320] Lustre: server umount lustre-MDT0001 complete [ 8538.458505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8553.818768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8554.053559] LustreError: 165712:0:(llog.c:1655:llog_backup()) MGC192.168.201.120@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8554.058195] Lustre: 165712:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.120@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8554.916591] LustreError: 165717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8554.944061] LustreError: 165717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8614.881063] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 8614.896269] Lustre: 165712:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.120@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 8620.516345] LustreError: 165717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8620.533923] LustreError: 165717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 50 previous similar messages [ 8655.870460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8660.108750] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8660.976498] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8670.211508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8671.206251] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2ec0854342a12005 [ 8671.224498] Lustre: MGC192.168.201.120@tcp: Connection restored to 0@lo (at 0@lo) [ 8676.860961] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8677.197822] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8690.021240] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8693.283886] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8704.475992] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8707.223931] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8738.272425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8738.275081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8738.292061] Lustre: Skipped 1 previous similar message [ 8738.307975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8738.309848] Lustre: Skipped 4 previous similar messages [ 8741.233666] Lustre: server umount lustre-MDT0000 complete [ 8744.939713] LustreError: 161847:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785396296 with bad export cookie 3368839045200289797 [ 8744.958329] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8745.367596] Lustre: server umount lustre-MDT0001 complete [ 8761.507677] Lustre: server umount lustre-OST0000 complete [ 8786.492403] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 8800.354372] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 8840.909930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8841.247672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8841.346180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8841.449711] Lustre: lustre-MDT0000: new disk, initializing [ 8841.506800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8841.511508] Lustre: Skipped 1 previous similar message [ 8841.521779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8848.005700] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8864.107501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8864.235462] Lustre: 170827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8864.464642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8873.118225] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8886.981975] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8894.358719] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8901.821279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8902.158305] Lustre: lustre-OST0000: new disk, initializing [ 8902.161805] Lustre: Skipped 1 previous similar message [ 8902.164695] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8902.168441] Lustre: Skipped 2 previous similar messages [ 8902.174558] Lustre: 172798:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8908.392701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8908.405172] Lustre: Skipped 1 previous similar message [ 8908.422764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8908.617629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8911.154741] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 8922.940538] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8928.812870] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8929.072928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8935.859891] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8936.150473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8939.506764] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8939.507602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8939.521526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8939.528113] LustreError: Skipped 1 previous similar message [ 8939.560106] Lustre: Skipped 2 previous similar messages [ 8953.311413] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8953.629650] Lustre: server umount lustre-OST0000 complete [ 8964.922768] Lustre: server umount lustre-MDT0000 complete [ 8965.088420] LustreError: 170838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8965.111197] LustreError: 170838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [ 8969.114861] LustreError: 170817:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785396520 with bad export cookie 3368839045200306632 [ 8969.125983] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8969.128912] LustreError: 170817:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8969.472119] Lustre: server umount lustre-MDT0001 complete [ 8979.950751] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:28:50 (1785396530) [ 8987.775568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8996.544931] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9009.965580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9016.516745] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9027.263398] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9035.170186] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9044.388440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9054.720653] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9067.414629] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9080.000186] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:30:29 (1785396629) [ 9090.020049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9090.023726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9090.029154] Lustre: Skipped 2 previous similar messages [ 9090.091530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9090.112062] Lustre: Skipped 7 previous similar messages [ 9094.585506] Lustre: server umount lustre-OST0000 complete [ 9102.821024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9108.577256] Lustre: server umount lustre-MDT0000 complete [ 9113.915422] LustreError: 175195:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785396665 with bad export cookie 3368839045200307514 [ 9113.929594] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9113.933947] LustreError: 175195:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9114.364915] Lustre: server umount lustre-MDT0001 complete [ 9130.511995] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 9133.940799] Key type lgssc unregistered [ 9134.438386] LNet: 178950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9134.445237] LNetError: 178950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9134.474274] LNet: Removed LNI 192.168.201.120@tcp [ 9135.918579] Key type .llcrypt unregistered [ 9135.923237] Key type ._llcrypt unregistered [ 9163.145864] Key type ._llcrypt registered [ 9163.148369] Key type .llcrypt registered [ 9163.280788] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9164.957471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9165.225211] alg: No test for adler32 (adler32-zlib) [ 9166.426500] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 9166.932927] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 9168.767687] Key type lgssc registered [ 9170.526726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9183.310783] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9183.343222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9199.072211] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9204.191443] LustreError: 180194:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.120@tcp: failed processing log, type 4: rc = -110 [ 9209.311423] LustreError: 180194:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 9209.330810] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 9209.343593] LustreError: MGC192.168.201.120@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 9209.362845] LustreError: 180194:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 9209.381525] LustreError: 180194:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 9209.390124] LustreError: 180194:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9209.533392] Lustre: server umount lustre-OST0000 complete [ 9209.537086] LustreError: 180194:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9221.724027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9223.497770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9228.583949] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9232.868543] LustreError: 180693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9232.896532] LustreError: 180693:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9237.994695] LustreError: 180694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9243.119486] LustreError: 180693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9245.355321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9245.983538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9251.711610] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9260.722711] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9269.362284] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9280.620394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9281.419669] LustreError: 182512:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 9281.444047] Lustre: 182512:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.120@tcp: invalid local config log lustre-OST0000: rc = -22 [ 9281.730867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9293.657751] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9306.690310] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9312.229694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9312.238400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9312.256159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9313.761483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9313.763079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9313.792922] Lustre: Skipped 1 previous similar message [ 9313.799407] Lustre: Skipped 1 previous similar message [ 9316.649913] Lustre: server umount lustre-MDT0000 complete [ 9318.899733] LustreError: 183140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9318.937863] LustreError: 183140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9322.941094] LustreError: 180673:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785396874 with bad export cookie 6305964571264307719 [ 9322.945349] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9322.986182] LustreError: 180673:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9323.844838] Lustre: server umount lustre-MDT0001 complete [ 9340.846168] Lustre: server umount lustre-OST0000 complete [ 9354.220056] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 9368.989623] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9421.247635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9421.831073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9421.893523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9422.107764] Lustre: lustre-MDT0000: new disk, initializing [ 9422.251171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9422.300636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9428.635955] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9441.761445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9441.994182] Lustre: 186044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9442.135169] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9442.138762] Lustre: Skipped 1 previous similar message [ 9442.235921] Lustre: lustre-MDT0001: new disk, initializing [ 9442.436821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9442.489159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9442.522376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9448.648074] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9459.781381] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9468.557449] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9480.180696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9480.954085] Lustre: lustre-OST0000: new disk, initializing [ 9480.964509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9480.983566] Lustre: 188016:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9481.248957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9490.367025] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9490.565977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9490.577137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9490.808726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9502.256410] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9509.659246] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9510.031622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9516.422952] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9516.665877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9521.635110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9521.637447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9521.643702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9521.643710] Lustre: Skipped 1 previous similar message [ 9521.665510] Lustre: Skipped 1 previous similar message [ 9525.092749] Lustre: server umount lustre-OST0000 complete [ 9531.878802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9531.880458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9531.898208] Lustre: Skipped 1 previous similar message [ 9531.902016] Lustre: Skipped 1 previous similar message [ 9536.608587] Lustre: server umount lustre-MDT0000 complete [ 9536.992968] LustreError: 187414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9537.019804] LustreError: 187414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9542.452054] LustreError: 186037:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397094 with bad export cookie 6305964571264308447 [ 9542.476042] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9542.483722] LustreError: 186037:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9543.074798] Lustre: server umount lustre-MDT0001 complete [ 9558.406973] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:38:28 (1785397108) [ 9582.577560] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [ 9586.407529] Key type lgssc unregistered [ 9587.065725] LNet: 190963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9587.083277] LNetError: 190963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9587.102746] LNet: Removed LNI 192.168.201.120@tcp [ 9588.797797] Key type .llcrypt unregistered [ 9588.803434] Key type ._llcrypt unregistered [ 9616.048653] Key type ._llcrypt registered [ 9616.049848] Key type .llcrypt registered [ 9616.198351] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9617.707418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9617.739933] alg: No test for adler32 (adler32-zlib) [ 9619.090965] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [ 9619.321948] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [ 9620.967162] Key type lgssc registered [ 9622.613939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9634.547956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9634.605652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9636.208073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9643.512075] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9645.560841] LustreError: 192187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9645.594724] LustreError: 192187:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9650.659152] LustreError: 192186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9653.830961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9654.233971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9661.296880] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9671.853364] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9679.699518] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9689.799271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9689.974532] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9689.980248] LustreError: 194051:0:(llog.c:1696:llog_backup()) MGC192.168.201.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 9689.988296] Lustre: 194051:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.120@tcp: can't backup local config lustre-OST0000: rc = -28 [ 9690.203891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9698.789526] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9707.290989] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9710.564956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9710.568425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9710.582528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9711.077349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9711.097840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9711.100440] Lustre: Skipped 1 previous similar message [ 9716.199758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9716.204135] Lustre: Skipped 1 previous similar message [ 9721.315283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9721.326846] Lustre: Skipped 3 previous similar messages [ 9724.383332] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9724.714129] Lustre: server umount lustre-MDT0000 complete [ 9726.439519] LustreError: 192914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.465381] LustreError: 192914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9729.836790] LustreError: 192166:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397281 with bad export cookie 4210840740018771511 [ 9729.840622] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9729.855494] LustreError: 192166:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9730.241368] Lustre: server umount lustre-MDT0001 complete [ 9745.452123] Lustre: server umount lustre-OST0000 complete [ 9761.573942] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [ 9780.385640] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [ 9831.841714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9832.118636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9832.170437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9832.278209] Lustre: lustre-MDT0000: new disk, initializing [ 9832.367740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9832.383700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9838.474671] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9853.675635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9853.837849] Lustre: 197605:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9853.905484] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9853.911656] Lustre: Skipped 1 previous similar message [ 9853.918936] LustreError: 198387:0:(llog.c:1696:llog_backup()) MGC192.168.201.120@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 9853.930880] LustreError: 198387:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 9853.935811] Lustre: 198387:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.120@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 9853.967855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9853.979211] Lustre: Skipped 1 previous similar message [ 9854.101399] Lustre: lustre-MDT0001: new disk, initializing [ 9854.268781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9854.301800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9854.311491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9859.821513] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9872.397707] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9881.108200] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9889.595421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9889.916623] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9889.919759] Lustre: Skipped 1 previous similar message [ 9889.924840] LustreError: 199550:0:(llog.c:1696:llog_backup()) MGC192.168.201.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 9889.935799] LustreError: 199550:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 9889.946839] Lustre: 199550:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.120@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 9889.955879] Lustre: 199550:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 9889.981683] Lustre: lustre-OST0000: new disk, initializing [ 9889.985322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9889.990706] Lustre: 199574:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9890.083092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9897.619190] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [ 9900.099597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9900.105097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9900.237599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9910.069668] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9916.582369] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9916.898236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9923.442930] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9923.802324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9931.237222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9931.238858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9931.248345] LustreError: Skipped 1 previous similar message [ 9931.249162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9931.273963] Lustre: Skipped 2 previous similar messages [ 9932.432765] Lustre: server umount lustre-OST0000 complete [ 9941.478123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9941.489930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9941.491049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9941.491054] Lustre: Skipped 1 previous similar message [ 9941.492049] Lustre: Skipped 1 previous similar message [ 9943.564726] Lustre: server umount lustre-MDT0000 complete [ 9946.593366] LustreError: 197611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9946.614346] LustreError: 197611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9948.566526] LustreError: 197596:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397500 with bad export cookie 4210840740018772225 [ 9948.568742] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9948.581863] LustreError: 197596:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9949.238665] Lustre: server umount lustre-MDT0001 complete [ 9961.226156] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:45:11 (1785397511) [ 9986.913483] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10001.888625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10002.238811] LustreError: 202726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10002.281192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10007.807641] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10011.630763] LustreError: 202727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10011.663249] LustreError: 202727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10020.713501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10028.615435] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10032.693814] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10041.871078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10042.397806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10042.400193] Lustre: Skipped 1 previous similar message [10050.316399] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10062.492389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10062.833423] Lustre: lustre-OST0001: new disk, initializing [10062.841873] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10062.857860] Lustre: 205040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10069.677710] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10071.195093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10071.208183] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10071.389619] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10085.004795] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10096.654115] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10099.499794] Lustre: DEBUG MARKER: ADD OST3 [10106.160791] loop: module loaded [10106.301128] loop0: detected capacity change from 0 to 204800000 [10106.598225] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10106.624315] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10106.653207] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10114.647678] Lustre: DEBUG MARKER: STOP OST3 [10114.948233] loop0: detected capacity change from 0 to 204800000 [10121.689205] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10121.960545] Lustre: lustre-OST0003: new disk, initializing [10121.971502] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [10121.982514] Lustre: 207088:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10122.211579] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [10122.230652] Lustre: Skipped 1 previous similar message [10123.232387] Lustre: *** cfs_fail_loc=2109, val=2*** [10123.404050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [10123.420342] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [10123.475783] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [10125.280086] LustreError: 207177:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [10125.774541] Lustre: Failing over lustre-OST0003 [10126.131497] Lustre: server umount lustre-OST0003 complete [10128.354077] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10128.357436] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10128.372136] LustreError: 204217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10128.402057] LustreError: 204217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10129.705542] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10142.284425] Lustre: DEBUG MARKER: START OST3 again [10153.986530] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10154.856332] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [10156.056401] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [10156.878912] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10156.899590] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [10163.257297] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10188.052931] Lustre: Failing over lustre-OST0003 [10188.222022] Lustre: server umount lustre-OST0003 complete [10190.847722] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10190.863942] Lustre: Skipped 1 previous similar message [10190.874317] LustreError: 209524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10190.893249] LustreError: 209524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [10210.814370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10210.832620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10210.862074] Lustre: Skipped 1 previous similar message [10210.875283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10210.879877] Lustre: Skipped 1 previous similar message [10218.000842] Lustre: server umount lustre-MDT0000 complete [10223.082151] LustreError: 202706:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397774 with bad export cookie 4210840740018773114 [10223.093902] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10230.656957] Lustre: server umount lustre-MDT0001 complete [10237.239788] Lustre: server umount lustre-OST0000 complete [10243.424537] Lustre: server umount lustre-OST0001 complete [10257.198613] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [10269.389168] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10323.611226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10324.198102] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10324.291735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10324.406190] Lustre: lustre-MDT0000: new disk, initializing [10324.515791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10324.528260] Lustre: Skipped 1 previous similar message [10324.553708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10330.718295] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10346.500272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10346.722455] Lustre: 212510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10346.777198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10346.780566] Lustre: Skipped 1 previous similar message [10346.872676] Lustre: lustre-MDT0001: new disk, initializing [10347.066317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10347.080084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10353.988526] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10367.457981] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10376.554807] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10387.978031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10388.430931] Lustre: lustre-OST0000: new disk, initializing [10388.448809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10388.470772] Lustre: 214481:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10389.828192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10389.836956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10389.909789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10397.847550] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10407.650587] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10413.660793] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10413.877236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10419.715712] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10420.177493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10425.826949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10425.838808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10425.849631] Lustre: Skipped 5 previous similar messages [10425.858711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10425.915651] Lustre: Skipped 13 previous similar messages [10437.087813] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10437.594850] Lustre: server umount lustre-OST0000 complete [10444.262652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10448.648521] Lustre: server umount lustre-MDT0000 complete [10451.428344] LustreError: 214184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10451.465100] LustreError: 214184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [10454.037058] LustreError: 212502:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398005 with bad export cookie 4210840740018775403 [10454.056755] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10454.057515] LustreError: 212502:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [10454.625643] Lustre: server umount lustre-MDT0001 complete [10459.456333] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:53:29 (1785398009) [10481.471878] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [10494.551092] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10543.439205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10544.007849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10544.066616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10544.217222] Lustre: lustre-MDT0000: new disk, initializing [10544.390144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10551.267189] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10564.973855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10565.314158] Lustre: 219098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10565.862707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10571.568420] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10586.129758] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10593.684676] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10603.186091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10603.651517] Lustre: 221068:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10603.797562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10603.800021] Lustre: Skipped 4 previous similar messages [10605.394195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10605.608024] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10612.711765] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10624.497578] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10629.748739] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10629.974235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10636.729332] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10637.063876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10641.381045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10641.385789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10641.389874] LustreError: Skipped 1 previous similar message [10641.390679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10641.390683] Lustre: Skipped 7 previous similar messages [10641.456201] Lustre: Skipped 4 previous similar messages [10645.786824] Lustre: server umount lustre-OST0000 complete [10658.143681] Lustre: server umount lustre-MDT0000 complete [10661.859623] LustreError: 219110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10661.873431] LustreError: 219110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10663.142442] LustreError: 219894:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398214 with bad export cookie 4210840740018776285 [10663.147938] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10663.160238] LustreError: 219894:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [10663.862521] Lustre: server umount lustre-MDT0001 complete [10676.153151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10683.649649] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10695.921247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10704.012277] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10715.200640] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10723.777269] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10733.313536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10743.407708] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [10753.997125] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10766.817533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10771.414187] Lustre: server umount lustre-OST0000 complete [10777.057258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10777.070458] Lustre: Skipped 5 previous similar messages [10777.087799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10777.089994] Lustre: Skipped 9 previous similar messages [10782.808287] Lustre: server umount lustre-MDT0000 complete [10787.683526] LustreError: 223266:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398339 with bad export cookie 4210840740018777167 [10787.698424] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10787.715720] LustreError: 223266:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 3 previous similar messages [10788.216140] Lustre: server umount lustre-MDT0001 complete [10896.311552] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [10900.140610] Key type lgssc unregistered [10900.502158] LNet: 226952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10900.519318] LNetError: 226952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10900.546067] LNet: Removed LNI 192.168.201.120@tcp [10902.286502] Key type .llcrypt unregistered [10902.291306] Key type ._llcrypt unregistered [10944.734360] Key type ._llcrypt registered [10944.738440] Key type .llcrypt registered [10944.900939] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [10966.209985] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [10967.953379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10967.996102] alg: No test for adler32 (adler32-zlib) [10969.115507] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [10969.385967] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [10971.079167] Key type lgssc registered [10972.418479] Lustre: Echo OBD driver; http://www.lustre.org/ [11013.797262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11013.832491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11015.081382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11015.125659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11015.225557] Lustre: lustre-MDT0000: new disk, initializing [11015.302358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11015.319715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11021.216805] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11033.947038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11034.109788] Lustre: 230324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11034.176810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11034.180794] Lustre: Skipped 1 previous similar message [11034.306595] Lustre: lustre-MDT0001: new disk, initializing [11034.426624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11034.450192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11034.482826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11040.065588] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11051.985658] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11062.902805] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11072.810088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11073.025510] Lustre: lustre-OST0000: new disk, initializing [11073.028113] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11073.032502] Lustre: 232294:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11073.116249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11080.376207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11080.397953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11080.728163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11083.621330] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11097.994794] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11105.798492] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11106.145541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11112.599360] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11113.076723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11121.633778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11121.644131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11121.653686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11126.775258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11126.787036] Lustre: Skipped 2 previous similar messages [11131.362390] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11131.634640] Lustre: server umount lustre-OST0000 complete [11136.407194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11136.426455] Lustre: Skipped 1 previous similar message [11136.437311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11141.604318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11141.611037] Lustre: Skipped 2 previous similar messages [11143.045562] Lustre: server umount lustre-MDT0000 complete [11147.585238] LustreError: 230316:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398699 with bad export cookie 13609207287391861460 [11147.594482] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11147.609876] LustreError: 230316:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [11147.986428] Lustre: server umount lustre-MDT0001 complete [11176.512746] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [11190.409493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11190.995630] LustreError: 235255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11191.025880] LustreError: 235255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11191.082780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11196.406775] LustreError: 235256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11196.436091] LustreError: 235256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11196.844505] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11200.491623] LustreError: 235255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11205.635320] LustreError: 235256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11210.726753] LustreError: 235255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11214.367889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11215.207774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11221.766433] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11226.340282] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11234.516485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11234.989475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11242.210891] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11253.208477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11253.399322] Lustre: lustre-OST0001: new disk, initializing [11253.402487] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11253.407209] Lustre: 237566:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11253.501125] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11261.163721] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11262.637069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11262.649749] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11262.842672] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11275.437316] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11284.205989] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11301.380210] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 04:07:31 (1785398851) [11306.977585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11306.983782] LustreError: Skipped 1 previous similar message [11306.985841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11307.003383] Lustre: Skipped 1 previous similar message [11307.012940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11309.039586] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11309.050673] Lustre: Skipped 1 previous similar message [11310.540631] Lustre: server umount lustre-MDT0000 complete [11314.146090] LustreError: 235256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11315.236170] LustreError: 237564:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398866 with bad export cookie 13609207287391862342 [11315.254908] LustreError: 237564:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [11315.257771] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11316.119180] Lustre: server umount lustre-MDT0001 complete [11330.795179] Lustre: server umount lustre-OST0000 complete [11334.623581] Lustre: 228547:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785398870/real 1785398870] req@ffffa0ba83001500 x1872126038635904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785398886 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11334.667248] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11334.682926] Lustre: Skipped 1 previous similar message [11335.571547] Lustre: server umount lustre-OST0001 complete [11345.864586] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [11363.010379] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [11407.278562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11407.591819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11407.614843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11407.700773] Lustre: lustre-MDT0000: new disk, initializing [11407.759541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11407.777776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11413.789143] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11430.965203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11431.151143] Lustre: 241534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11431.275899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11431.280737] Lustre: Skipped 1 previous similar message [11431.489714] Lustre: lustre-MDT0001: new disk, initializing [11431.785473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11431.804310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11438.062263] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11449.021636] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11455.975704] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11464.857375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11465.337768] Lustre: lustre-OST0000: new disk, initializing [11465.350218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11465.356411] Lustre: 243502:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11466.461793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11466.477221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11466.602020] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11473.071235] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11486.010564] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11491.986779] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11492.186628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11498.434745] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11498.774605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11502.560631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11502.569907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11502.581891] LustreError: Skipped 1 previous similar message [11502.591804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11502.614469] Lustre: Skipped 1 previous similar message [11502.619241] Lustre: Skipped 4 previous similar messages [11516.383144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11516.828235] Lustre: server umount lustre-OST0000 complete [11522.442911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11522.450831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11522.457360] Lustre: Skipped 1 previous similar message [11522.483468] Lustre: Skipped 6 previous similar messages [11528.909847] Lustre: server umount lustre-MDT0000 complete [11532.773562] LustreError: 244845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11532.783065] LustreError: 244845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11534.546266] LustreError: 241526:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785399086 with bad export cookie 13609207287391864477 [11534.555186] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11534.560592] LustreError: 241526:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [11535.027240] Lustre: server umount lustre-MDT0001 complete [11548.423706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11549.245990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11549.253357] Lustre: Skipped 2 previous similar messages [11554.834194] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11567.714113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11574.780421] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11585.734233] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11592.918910] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11602.176543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11612.170476] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [11623.144951] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11771.872491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11771.875628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11771.892456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11771.897933] Lustre: Skipped 2 previous similar messages [11785.204914] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11785.443439] Lustre: server umount lustre-OST0000 complete [11793.377091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11805.151756] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11807.337808] Lustre: server umount lustre-MDT0000 complete [11808.737045] LustreError: 248247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11808.764879] LustreError: 248247:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11813.354754] LustreError: 247571:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785399364 with bad export cookie 13609207287391865359 [11813.360470] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11813.373343] LustreError: 247571:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [11814.689677] Lustre: server umount lustre-MDT0001 complete [11891.872340] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [11895.220604] Key type lgssc unregistered [11895.558769] LNet: 249338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11895.565340] LNetError: 249338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11895.596247] LNet: Removed LNI 192.168.201.120@tcp [11896.786165] Key type .llcrypt unregistered [11896.787542] Key type ._llcrypt unregistered [11928.055427] Key type ._llcrypt registered [11928.057273] Key type .llcrypt registered [11928.225436] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [11951.714913] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [11953.356654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11953.397360] alg: No test for adler32 (adler32-zlib) [11954.937555] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [11955.265533] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [11957.568980] Key type lgssc registered [11959.250465] Lustre: Echo OBD driver; http://www.lustre.org/ [12023.255136] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [12036.817392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12036.854320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12038.222227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12038.274787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12038.403392] Lustre: lustre-MDT0000: new disk, initializing [12038.519730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12038.550241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12043.559764] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12057.295550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12057.462799] Lustre: 253436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12057.571353] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12057.577322] Lustre: Skipped 1 previous similar message [12057.710741] Lustre: lustre-MDT0001: new disk, initializing [12057.885578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12057.965545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12057.975177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12064.102463] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12069.062767] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12077.865874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12078.352508] Lustre: lustre-OST0000: new disk, initializing [12078.360721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12078.365792] Lustre: 255069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12078.526924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12086.419851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12086.440603] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12086.605443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12087.338909] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12103.410420] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12103.541827] Lustre: lustre-OST0001: new disk, initializing [12103.544836] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12103.552123] Lustre: 255940:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12103.668589] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12110.653363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12110.663569] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12110.890991] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12115.567704] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12130.311417] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12139.334661] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12155.360542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12155.372122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12155.403178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12156.920540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12156.923197] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12156.960603] Lustre: Skipped 1 previous similar message [12162.107228] Lustre: server umount lustre-MDT0000 complete [12167.079313] LustreError: 254240:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785399718 with bad export cookie 3577121836098748301 [12167.088813] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12167.093412] LustreError: 254240:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [12167.143398] LustreError: 256567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12167.147170] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12167.161166] LustreError: 256567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [12167.194267] Lustre: Skipped 1 previous similar message [12167.205262] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12167.215018] Lustre: Skipped 4 previous similar messages [12167.850163] Lustre: server umount lustre-MDT0001 complete [12174.026576] Lustre: server umount lustre-OST0000 complete [12178.958892] Lustre: server umount lustre-OST0001 complete [12197.395554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12210.317566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12220.676800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [12234.038679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [12257.292984] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [12269.801771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12269.841938] Lustre: lustre-MDT0000: reset Object Index mappings [12270.274265] LustreError: 260957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12270.378695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12275.685951] LustreError: 260958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12276.200484] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12279.790925] LustreError: 260957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12284.900566] LustreError: 260958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12286.718317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12286.740344] Lustre: lustre-MDT0001: reset Object Index mappings [12287.516798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12293.695312] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12297.309499] Lustre: 262100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12304.830091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12305.454619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12311.593411] LustreError: 262453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12313.692018] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12314.682145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [12324.912503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12330.537363] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [12335.883466] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12345.281326] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12350.091038] Lustre: 263974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12359.137405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12359.153602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12359.167129] Lustre: Skipped 1 previous similar message [12359.180039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12363.240790] Lustre: server umount lustre-MDT0000 complete [12366.310581] LustreError: 260953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12366.323519] LustreError: 260953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [12367.993672] LustreError: 260938:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785399919 with bad export cookie 3577121836098755091 [12367.998545] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12368.001380] LustreError: 260938:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [12368.376543] Lustre: server umount lustre-MDT0001 complete [12383.441753] Lustre: server umount lustre-OST0000 complete [12386.784602] Lustre: 250919:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785399922/real 1785399922] req@ffffa0ba8e819880 x1872127072548608/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785399938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12386.812725] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12386.828859] Lustre: Skipped 3 previous similar messages [12388.652421] Lustre: server umount lustre-OST0001 complete [12412.331789] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 04:26:02 (1785399962) [12433.599192] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [12450.745216] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [12497.202381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12497.728310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12497.793979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12497.951758] Lustre: lustre-MDT0000: new disk, initializing [12498.085713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12498.095398] Lustre: Skipped 1 previous similar message [12498.113682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12505.022913] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12521.333909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12521.553127] Lustre: 268249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12521.668275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12521.670414] Lustre: Skipped 1 previous similar message [12521.814861] Lustre: lustre-MDT0001: new disk, initializing [12522.028975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12522.043966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12529.454737] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12542.384533] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12551.680825] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12565.427248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12565.903400] Lustre: lustre-OST0000: new disk, initializing [12565.907570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12565.916749] Lustre: 270219:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12566.071554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12566.079181] Lustre: Skipped 1 previous similar message [12567.052407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12567.070378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12567.284992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12574.900376] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12585.656597] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12591.810220] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12592.036113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12597.541937] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12597.809406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12603.361332] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12603.364942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12603.368220] LustreError: Skipped 1 previous similar message [12603.375152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12603.383753] Lustre: Skipped 1 previous similar message [12603.408015] Lustre: Skipped 5 previous similar messages [12606.601772] Lustre: server umount lustre-OST0000 complete [12613.606856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12618.474976] Lustre: server umount lustre-MDT0000 complete [12618.720300] LustreError: 268259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12618.746870] LustreError: 268259:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12622.971036] LustreError: 268240:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785400174 with bad export cookie 3577121836098757184 [12622.973391] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12622.985455] LustreError: 268240:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [12623.836775] Lustre: server umount lustre-MDT0001 complete [12652.219547] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [12666.174584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12672.521673] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12685.139795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12691.534529] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12696.743884] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12705.929719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12706.349377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12706.352046] Lustre: Skipped 2 previous similar messages [12715.446989] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12727.452944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12727.730405] Lustre: lustre-OST0001: new disk, initializing [12727.736518] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12727.742896] Lustre: 275490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12729.789654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12729.795296] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12729.967183] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12737.222179] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12751.933760] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12759.446505] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12766.053187] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [12767.483225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [12775.918210] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12775.934435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12775.936187] Lustre: Skipped 3 previous similar messages [12775.950326] Lustre: Skipped 4 previous similar messages [12782.172427] Lustre: server umount lustre-MDT0000 complete [12786.149427] LustreError: 273178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12786.192191] LustreError: 273178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [12789.680606] LustreError: 273161:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785400341 with bad export cookie 3577121836098758066 [12789.693904] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12789.701114] LustreError: 273161:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [12791.123838] Lustre: server umount lustre-MDT0001 complete [12807.477360] Lustre: 250916:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785400342/real 1785400342] req@ffffa0ba8c3c4380 x1872127072670848/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785400358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12807.664630] Lustre: server umount lustre-OST0000 complete [12809.696375] Lustre: 250918:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785400345/real 1785400345] req@ffffa0ba8c3c7b80 x1872127072671104/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785400361 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12813.327231] Lustre: server umount lustre-OST0001 complete [12835.482807] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [12850.003138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12858.688986] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12872.583411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12880.218060] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12883.984697] Lustre: 279692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12892.245537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12895.936441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [12903.373337] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12916.295521] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12921.889917] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [12924.846926] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [12935.983810] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12941.649185] Lustre: 281569:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12947.257479] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [12947.463778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [12961.186526] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 04:35:11 (1785400511) [12978.155245] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12978.179845] Lustre: Skipped 3 previous similar messages [12978.190972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12978.216919] Lustre: Skipped 7 previous similar messages [12984.462852] Lustre: server umount lustre-MDT0000 complete [12988.399260] LustreError: 279605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12988.409460] LustreError: 279605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [12989.275529] LustreError: 278528:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785400540 with bad export cookie 3577121836098764611 [12989.313574] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12996.047902] Lustre: server umount lustre-MDT0001 complete [13007.596411] Lustre: server umount lustre-OST0000 complete [13013.073792] Lustre: server umount lustre-OST0001 complete [13024.010504] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [13040.253543] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [13106.488543] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [13126.050225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13126.294910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13126.315265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13126.400405] Lustre: lustre-MDT0000: new disk, initializing [13126.495792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13126.499321] Lustre: Skipped 5 previous similar messages [13126.511462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13132.414847] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13145.514505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13145.635470] Lustre: 285866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13145.966784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13151.023413] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13166.894691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13167.343199] Lustre: 287497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13169.380840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13178.195830] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13191.130643] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13191.378291] Lustre: 288368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13193.071060] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13193.081790] Lustre: Skipped 1 previous similar message [13193.243383] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13199.481870] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13213.263891] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13219.826547] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [13219.840385] Lustre: Skipped 1 previous similar message [13247.269952] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 04:39:57 (1785400797) [13253.602133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13253.615722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13253.632378] Lustre: Skipped 5 previous similar messages [13253.642997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13253.647528] Lustre: Skipped 8 previous similar messages [13256.719841] Lustre: server umount lustre-MDT0000 complete [13259.750294] LustreError: 285871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13259.781400] LustreError: 285871:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [13261.246984] LustreError: 285857:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785400812 with bad export cookie 3577121836098772248 [13261.269955] LustreError: 285857:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [13261.270975] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13261.660903] Lustre: server umount lustre-MDT0001 complete [13276.869556] Lustre: server umount lustre-OST0000 complete [13292.197272] Lustre: server umount lustre-OST0001 complete [13307.546569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13313.964673] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13324.257501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13330.028877] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13340.077645] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13349.211973] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13359.916495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13367.760700] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13377.071930] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13412.750502] loop1: detected capacity change from 0 to 204800000 [13413.232880] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13413.255089] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13413.281288] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13420.150702] loop1: detected capacity change from 0 to 204800000 [13426.196162] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13426.314630] LustreError: 290606:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs18413: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [13426.335900] LustreError: 290606:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs18413 [13426.344668] LustreError: 290606:0:(mgs_handler.c:517:mgs_target_reg()) fs18413-MDT0001: Error checking target index: rc = -13 [13426.366961] LustreError: fs18413-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [13426.373823] LustreError: 294361:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [13426.382810] LustreError: 294361:0:(tgt_mount.c:2129:server_put_super()) no obd fs18413-MDT0001 [13426.390905] LustreError: 294361:0:(tgt_mount.c:116:server_deregister_mount()) fs18413-MDT0001 not registered [13426.427150] Lustre: server umount fs18413-MDT0001 complete [13426.436646] LustreError: 294361:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [13436.194093] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13436.292756] Lustre: Setting parameter fs18413-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs18413-MDT0001 [13436.336521] Lustre: srv-fs18413-MDT0001: No data found on store. Initialize space. [13436.344266] Lustre: Skipped 4 previous similar messages [13436.497624] Lustre: fs18413-MDT0001: new disk, initializing [13436.501208] Lustre: Skipped 3 previous similar messages [13436.735482] Lustre: srv-fs18413-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13436.743385] Lustre: 294828:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18413-MDT0001-mdtlov: can't create new llog: rc = -115 [13436.760427] LustreError: 294828:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18413-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [13441.066869] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13449.362339] Lustre: server umount fs18413-MDT0001 complete [13450.777403] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [13455.847440] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [13469.836885] loop1: detected capacity change from 0 to 204800000 [13479.848140] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13481.036472] Lustre: 296397:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18413-MDT0001-mdtlov: can't create new llog: rc = -115 [13481.063659] LustreError: 296397:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18413-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [13488.263710] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13494.688703] Lustre: server umount fs18413-MDT0001 complete [13507.046547] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13507.060483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13507.067423] Lustre: Skipped 4 previous similar messages [13507.088558] Lustre: Skipped 6 previous similar messages [13521.383723] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13522.085421] Lustre: server umount lustre-MDT0000 complete [13535.981334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13536.472173] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13541.880883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13543.688599] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13554.266861] loop1: detected capacity change from 0 to 204800000 [13560.283676] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13560.908724] Lustre: 298845:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18413-MDT0001-mdtlov: can't create new llog: rc = -115 [13560.922212] LustreError: 298845:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18413-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [13567.018381] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13572.292238] Lustre: server umount fs18413-MDT0001 complete [13603.299147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13607.472527] Lustre: server umount lustre-OST0000 complete [13613.537148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13613.544120] LustreError: Skipped 1 previous similar message [13626.335160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13627.053618] Lustre: server umount lustre-MDT0000 complete [13632.847222] LustreError: 290601:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785401184 with bad export cookie 3577121836098774929 [13632.852984] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13632.861493] LustreError: 290601:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [13633.888472] Lustre: server umount lustre-MDT0001 complete [13649.478131] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [13653.546443] Key type lgssc unregistered [13653.874970] LNet: 301194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13653.881246] LNetError: 301194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13653.902798] LNet: Removed LNI 192.168.201.120@tcp [13655.327219] Key type .llcrypt unregistered [13655.328615] Key type ._llcrypt unregistered [13681.382702] Key type ._llcrypt registered [13681.385473] Key type .llcrypt registered [13681.499149] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [13682.779522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13682.859599] alg: No test for adler32 (adler32-zlib) [13684.404285] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [13684.725760] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [13686.435049] Key type lgssc registered [13688.311363] Lustre: Echo OBD driver; http://www.lustre.org/ [13700.420614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13700.486745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13702.295935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13710.108744] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13711.855549] LustreError: 302468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13711.896487] LustreError: 302468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13716.999512] LustreError: 302469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13721.720979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13722.011650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13725.576694] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13732.821186] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13739.417344] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13746.030829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13746.384560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13747.472295] LustreError: 304299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13747.498165] LustreError: 304299:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13752.814068] LustreError: 304299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13752.827973] LustreError: 304299:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13752.957732] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13757.939440] LustreError: 304299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13757.961184] LustreError: 304299:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13762.443276] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13768.203665] LustreError: 304384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13768.227032] LustreError: 304384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13784.740423] LustreError: 304384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13784.791919] LustreError: 304384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [13788.360249] loop1: detected capacity change from 0 to 204800000 [13788.775470] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13788.801228] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13788.823977] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13796.079065] loop1: detected capacity change from 0 to 204800000 [13804.900656] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13805.248883] Lustre: fs2172-OST0000: new disk, initializing [13805.252191] Lustre: srv-fs2172-OST0000: No data found on store. Initialize space. [13805.259894] Lustre: 306209:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13805.352162] Lustre: fs2172-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13815.275365] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13819.372873] LustreError: 304301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13819.399961] LustreError: 304301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [13828.886355] Lustre: server umount fs2172-OST0000 complete [13829.624535] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [13834.740317] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [13844.921421] loop1: detected capacity change from 0 to 204800000 [13851.596582] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13852.089572] Lustre: fs2172-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13859.695278] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13870.192549] loop2: detected capacity change from 0 to 204800000 [13870.528430] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13870.545860] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13870.577634] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13875.427851] loop2: detected capacity change from 0 to 204800000 [13883.367551] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13883.562279] LustreError: fs2172-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [13883.582410] LustreError: fs2172-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [13883.601298] LustreError: 309247:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [13883.615517] LustreError: 309247:0:(tgt_mount.c:2129:server_put_super()) no obd fs2172-OST0001 [13883.634830] LustreError: 309247:0:(tgt_mount.c:116:server_deregister_mount()) fs2172-OST0001 not registered [13883.673647] Lustre: server umount fs2172-OST0001 complete [13883.675855] LustreError: 309247:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [13885.927835] LustreError: 304384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13885.967030] LustreError: 304384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [13900.305244] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13900.771796] Lustre: fs2172-OST0001: new disk, initializing [13900.778463] Lustre: srv-fs2172-OST0001: No data found on store. Initialize space. [13900.801780] Lustre: 309800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13900.987179] Lustre: fs2172-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13910.786345] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13921.679584] Lustre: server umount fs2172-OST0000 complete [13934.120613] Lustre: server umount fs2172-OST0001 complete [13953.350982] loop1: detected capacity change from 0 to 204800000 [13965.609432] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13966.207081] Lustre: fs2172-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13975.395604] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [13983.214057] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [13985.383086] loop2: detected capacity change from 0 to 204800000 [13988.366347] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [13994.455651] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14006.129943] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [14014.466837] LustreError: 304302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14014.480582] LustreError: 304302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 74 previous similar messages [14026.385275] Lustre: server umount fs2172-OST0000 complete [14042.235982] Lustre: server umount fs2172-OST0001 complete [14065.122822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14065.137802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14065.138574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14065.178863] LustreError: Skipped 1 previous similar message [14069.093686] Lustre: server umount lustre-OST0000 complete [14075.360385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14075.374133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14075.387688] Lustre: Skipped 1 previous similar message [14075.393484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14075.399070] Lustre: Skipped 1 previous similar message [14080.537231] Lustre: server umount lustre-MDT0000 complete [14085.983036] LustreError: 302448:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785401637 with bad export cookie 6752449468776841608 [14085.986678] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14085.996633] LustreError: 302448:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [14086.746325] Lustre: server umount lustre-MDT0001 complete [14102.306122] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [14106.202336] Key type lgssc unregistered [14106.873050] LNet: 315736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14106.885843] LNetError: 315736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14107.946488] LNet: Removed LNI 192.168.201.120@tcp [14109.359335] Key type .llcrypt unregistered [14109.362930] Key type ._llcrypt unregistered [14152.616986] Key type ._llcrypt registered [14152.619457] Key type .llcrypt registered [14152.781590] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [14174.819068] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [14177.840288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14177.900943] alg: No test for adler32 (adler32-zlib) [14179.668700] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [14180.791676] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [14182.426149] Key type lgssc registered [14184.475640] Lustre: Echo OBD driver; http://www.lustre.org/ [14242.766742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14242.814677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14244.056092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14244.077232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14244.127701] Lustre: lustre-MDT0000: new disk, initializing [14244.158593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14244.167762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14249.906759] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [14266.144381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14266.519538] Lustre: 319598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14266.812993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14266.828128] Lustre: Skipped 1 previous similar message [14267.243907] Lustre: lustre-MDT0001: new disk, initializing [14267.732343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14267.816695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14267.904238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14274.923769] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [14287.399918] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14295.756882] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14304.364637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14304.672616] Lustre: lustre-OST0000: new disk, initializing [14304.679775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14304.688352] Lustre: 321573:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14304.809733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14310.813176] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [14314.149278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14314.191257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14314.353781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14327.133412] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14331.544798] Lustre: Mounted lustre-client [14333.081155] Lustre: Unmounted lustre-client [14347.240802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14347.266115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14347.282855] Lustre: Skipped 1 previous similar message [14352.363729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14352.380872] Lustre: Skipped 1 previous similar message [14357.505232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14357.516480] Lustre: Skipped 1 previous similar message [14361.055149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14361.334606] Lustre: server umount lustre-OST0000 complete [14367.722568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14367.763734] Lustre: Skipped 1 previous similar message [14367.786341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14371.792983] Lustre: server umount lustre-MDT0000 complete [14376.727351] LustreError: 319592:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785401928 with bad export cookie 14847085945697923581 [14376.734876] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14376.736954] LustreError: 319592:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [14377.504243] Lustre: server umount lustre-MDT0001 complete [14387.807446] LNet: 1 local NIs in recovery (showing 1): 192.168.201.120@tcp [14392.043608] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [14396.047707] Key type lgssc unregistered [14396.321580] LNet: 323738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14396.331114] LNetError: 323738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14460.847066] LNet: Removed LNI 192.168.201.120@tcp [14462.464237] Key type .llcrypt unregistered [14462.467440] Key type ._llcrypt unregistered [14487.659678] Key type ._llcrypt registered [14487.663668] Key type .llcrypt registered [14487.771715] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [14510.940319] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [14513.052315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14513.219573] alg: No test for adler32 (adler32-zlib) [14514.517670] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [14514.819537] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [14516.567185] Key type lgssc registered [14518.031375] Lustre: Echo OBD driver; http://www.lustre.org/ [14560.928850] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 05:01:50 (1785402110) [14569.275780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14569.312409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14570.720205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14570.825595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14570.941921] Lustre: lustre-MDT0000: new disk, initializing [14571.111317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14571.133978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14576.959187] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [14588.138614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14588.305778] Lustre: 326974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14588.369785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14588.374601] Lustre: Skipped 1 previous similar message [14588.519599] Lustre: lustre-MDT0001: new disk, initializing [14588.645120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14588.678272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14588.689400] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14594.323209] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [14607.083149] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14616.589495] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14624.225503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14624.489897] Lustre: lustre-OST0000: new disk, initializing [14624.492830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14624.501791] Lustre: 328948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14624.607949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14631.019629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14631.031799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14631.170607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14632.199591] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [14645.655258] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14672.364783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14672.365406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14672.369310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14672.398533] Lustre: Skipped 1 previous similar message [14672.401987] LustreError: Skipped 1 previous similar message [14675.100154] Lustre: server umount lustre-OST0000 complete [14681.056284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14681.070700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14681.100306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14681.107618] Lustre: Skipped 1 previous similar message [14682.595188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14682.597065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14687.253549] Lustre: server umount lustre-MDT0000 complete [14692.836234] LustreError: 326986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14692.870157] LustreError: 326986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14693.436549] LustreError: 326968:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402244 with bad export cookie 13521462224196756099 [14693.445317] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14693.449198] LustreError: 326968:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [14694.061771] Lustre: server umount lustre-MDT0001 complete [14709.527761] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [14714.761311] Key type lgssc unregistered [14715.323475] LNet: 330858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14715.347322] LNetError: 330858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14716.394384] LNet: Removed LNI 192.168.201.120@tcp [14717.507492] Key type .llcrypt unregistered [14717.511110] Key type ._llcrypt unregistered [14745.156177] Key type ._llcrypt registered [14745.158435] Key type .llcrypt registered [14745.268723] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [14746.116597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14746.248510] alg: No test for adler32 (adler32-zlib) [14747.412171] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [14747.751089] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [14749.423174] Key type lgssc registered [14751.237760] Lustre: Echo OBD driver; http://www.lustre.org/ [14761.594729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14761.618770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14763.474763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14770.301189] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [14772.714499] LustreError: 332135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14772.742614] LustreError: 332135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14777.827651] LustreError: 332136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14782.107918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14783.007481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14793.636436] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [14810.716259] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14818.313621] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14828.354529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14828.786149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14838.362354] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [14850.342641] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14867.689652] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14870.313917] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14878.918435] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.20@tcp (stopping) [14879.201978] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14879.231831] Lustre: Skipped 1 previous similar message [14882.427326] Lustre: server umount lustre-MDT0000 complete [14884.002057] LustreError: 332132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14887.519912] LustreError: 332117:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402439 with bad export cookie 8382177270485903917 [14887.540911] LustreError: 332117:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [14887.542348] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14889.118505] LustreError: 332131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14889.151085] LustreError: 332131:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14889.164725] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.20@tcp (stopping) [14889.170922] Lustre: Skipped 3 previous similar messages [14889.444654] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14889.453666] Lustre: Skipped 1 previous similar message [14892.449043] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14892.456577] Lustre: Skipped 1 previous similar message [14894.107359] Lustre: server umount lustre-MDT0001 complete [14900.157691] Lustre: server umount lustre-OST0000 complete [14911.181256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14919.422964] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [14923.655595] Lustre: 336326:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.120@tcp [14927.488112] Lustre: server umount MGS complete [14939.899457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14940.927277] LustreError: 336899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14940.953961] LustreError: 336899:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14941.087965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14946.723027] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [14951.424517] LustreError: 336899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14951.449145] LustreError: 336899:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14959.862443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14960.487821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14966.243560] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [14979.714529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14980.405514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14990.793906] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [15001.717899] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15004.239518] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [15014.110645] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15017.129215] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15029.499286] Lustre: server umount lustre-OST0000 complete [15036.898307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15036.908559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15036.914074] Lustre: Skipped 1 previous similar message [15036.935482] Lustre: Skipped 1 previous similar message [15042.019859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15042.039268] Lustre: Skipped 1 previous similar message [15045.160484] Lustre: server umount lustre-MDT0000 complete [15047.143084] Lustre: 331448:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785402582/real 1785402582] req@000000009734dd42 x1872130001153920/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785402598 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15047.147096] LustreError: 337629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15047.172980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15047.217162] LustreError: 337629:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15048.159123] Lustre: 331445:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785402583/real 1785402583] req@000000001a7ff23c x1872130001154688/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785402599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15050.232659] LustreError: 336881:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402601 with bad export cookie 8382177270485904757 [15050.236793] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15050.248841] LustreError: 336881:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [15050.705098] Lustre: server umount lustre-MDT0001 complete [15066.495475] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [15069.909166] Key type lgssc unregistered [15070.380403] LNet: 340340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15070.387506] LNetError: 340340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15070.412030] LNet: Removed LNI 192.168.201.120@tcp [15071.817196] Key type .llcrypt unregistered [15071.822192] Key type ._llcrypt unregistered [15096.743615] Key type ._llcrypt registered [15096.750772] Key type .llcrypt registered [15096.935785] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [15117.060433] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [15118.109326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15118.276706] alg: No test for adler32 (adler32-zlib) [15119.514250] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [15119.830068] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [15121.551788] Key type lgssc registered [15123.007583] Lustre: Echo OBD driver; http://www.lustre.org/ [15168.651166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15168.746222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15170.305410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15170.355732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15170.535506] Lustre: lustre-MDT0000: new disk, initializing [15170.691199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15170.709813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15177.060433] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [15190.856059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15191.022361] Lustre: 343504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15191.081230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15191.083676] Lustre: Skipped 1 previous similar message [15191.208728] Lustre: lustre-MDT0001: new disk, initializing [15191.329614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15191.365629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15191.386377] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15199.030730] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [15211.901660] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15219.876514] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15230.125793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15230.678592] Lustre: lustre-OST0000: new disk, initializing [15230.682288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15230.691443] Lustre: 345478:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15230.906872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15239.734967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15239.752154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15239.895691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15241.513267] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [15252.850908] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15258.189847] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15258.396162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15264.979866] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15265.268617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15270.897610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15270.902353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15270.905166] LustreError: Skipped 1 previous similar message [15270.908838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15270.935980] Lustre: Skipped 1 previous similar message [15273.724832] Lustre: server umount lustre-OST0000 complete [15278.560862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15278.570673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15278.590390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15278.596017] Lustre: Skipped 1 previous similar message [15281.123483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15281.125741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15283.937614] Lustre: server umount lustre-MDT0000 complete [15288.524936] LustreError: 343496:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402840 with bad export cookie 13256681622219539950 [15288.534144] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15288.541903] LustreError: 343496:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 4 previous similar messages [15289.204461] Lustre: server umount lustre-MDT0001 complete [15302.127570] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 05:14:11 (1785402851) [15324.079596] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [15337.412575] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [15391.007910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15391.240038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15391.271561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15391.398174] Lustre: lustre-MDT0000: new disk, initializing [15391.520263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15391.540144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15397.159098] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [15410.409733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15410.474413] Lustre: 350245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15410.515508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15410.517384] Lustre: Skipped 1 previous similar message [15410.571150] Lustre: lustre-MDT0001: new disk, initializing [15410.662701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15410.732379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15410.751404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15417.242583] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [15429.990163] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15440.054675] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15449.373846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15449.812646] Lustre: lustre-OST0000: new disk, initializing [15449.819754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15449.832797] Lustre: 352216:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15450.016167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15457.377090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15457.405610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15457.591367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15459.902413] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [15471.463599] Lustre: DEBUG MARKER: oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15519.207696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15519.208584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15519.209231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15519.209236] Lustre: Skipped 1 previous similar message [15519.266936] Lustre: Skipped 1 previous similar message [15521.659720] Lustre: server umount lustre-OST0000 complete [15528.435169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15528.448888] LustreError: Skipped 1 previous similar message [15528.478083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15528.518588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15528.531799] Lustre: Skipped 1 previous similar message [15534.569854] LustreError: 350256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15534.594356] LustreError: 350256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15534.751489] Lustre: server umount lustre-MDT0000 complete [15538.668757] LustreError: 350250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15538.693506] LustreError: 350250:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15541.064651] LustreError: 350236:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785403092 with bad export cookie 13256681622219540832 [15541.070469] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15541.082929] LustreError: 350236:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [15542.081705] Lustre: server umount lustre-MDT0001 complete [15558.518407] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [15565.153944] Key type lgssc unregistered [15565.710932] LNet: 354397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15565.718717] LNetError: 354397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15566.759956] LNet: Removed LNI 192.168.201.120@tcp [15568.053905] Key type .llcrypt unregistered [15568.058209] Key type ._llcrypt unregistered [15695.964750] Key type ._llcrypt registered [15695.966810] Key type .llcrypt registered [15696.064791] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [15696.357642] Key type .llcrypt unregistered [15696.363224] Key type ._llcrypt unregistered [15807.999929] Key type ._llcrypt registered [15808.001340] Key type .llcrypt registered [15808.194799] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [15808.845129] Key type .llcrypt unregistered [15808.849472] Key type ._llcrypt unregistered [15853.792214] Key type ._llcrypt registered [15853.794404] Key type .llcrypt registered [15853.866375] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [15854.049104] Key type .llcrypt unregistered [15854.059965] Key type ._llcrypt unregistered [15900.231838] Key type ._llcrypt registered [15900.241365] Key type .llcrypt registered [15900.524954] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [15900.898456] Key type .llcrypt unregistered [15900.899654] Key type ._llcrypt unregistered [15991.433465] Key type ._llcrypt registered [15991.435517] Key type .llcrypt registered [15991.552625] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [15991.828662] Key type .llcrypt unregistered [15991.831454] Key type ._llcrypt unregistered [16028.793279] Key type ._llcrypt registered [16028.796220] Key type .llcrypt registered [16028.899490] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [16029.106679] Key type .llcrypt unregistered [16029.108342] Key type ._llcrypt unregistered [16067.180463] Key type ._llcrypt registered [16067.182738] Key type .llcrypt registered [16067.283963] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [16067.472380] Key type .llcrypt unregistered [16067.474226] Key type ._llcrypt unregistered [16102.303731] Key type ._llcrypt registered [16102.310891] Key type .llcrypt registered [16102.509499] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [16102.969356] Key type .llcrypt unregistered [16102.980093] Key type ._llcrypt unregistered [16139.465963] Key type ._llcrypt registered [16139.469988] Key type .llcrypt registered [16139.601250] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [16140.029761] Key type .llcrypt unregistered [16140.033203] Key type ._llcrypt unregistered [16176.096995] Key type ._llcrypt registered [16176.098514] Key type .llcrypt registered [16176.178265] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [16176.531603] Key type .llcrypt unregistered [16176.533220] Key type ._llcrypt unregistered [16270.640648] Key type ._llcrypt registered [16270.643417] Key type .llcrypt registered [16270.747416] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [16271.193396] Key type .llcrypt unregistered [16271.197456] Key type ._llcrypt unregistered [16311.258337] Key type ._llcrypt registered [16311.262471] Key type .llcrypt registered [16311.420590] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [16311.686278] Key type .llcrypt unregistered [16311.688226] Key type ._llcrypt unregistered [16355.809315] Key type ._llcrypt registered [16355.811563] Key type .llcrypt registered [16355.960431] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [16356.383459] Key type .llcrypt unregistered [16356.385685] Key type ._llcrypt unregistered [16428.565621] Key type ._llcrypt registered [16428.566828] Key type .llcrypt registered [16428.644355] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [16449.001399] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [16450.200571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16450.303738] alg: No test for adler32 (adler32-zlib) [16451.410895] Lustre: Lustre: Build Version: 2.17.56_2_g669939b [16451.533036] LNet: Added LNI 192.168.201.120@tcp [8/256/0/180] [16453.151236] Key type lgssc registered [16455.480309] Lustre: Echo OBD driver; http://www.lustre.org/ [16515.574473] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16515.658445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16517.216345] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [16517.278479] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [16517.418566] Lustre: tst250fs-MDT0000: new disk, initializing [16517.614574] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16517.662708] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16523.340535] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [16536.067697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16536.201198] Lustre: 374881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16536.228236] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [16536.229982] Lustre: Skipped 1 previous similar message [16536.302555] Lustre: tst250fs-MDT0001: new disk, initializing [16536.392360] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16536.434966] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16536.457239] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16541.451271] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [16546.320972] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [16555.072034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16555.406477] Lustre: tst250fs-OST0000: new disk, initializing [16555.409551] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [16555.417312] Lustre: 376514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16555.587486] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16557.345209] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16557.360309] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16557.595811] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16564.714972] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [16579.519222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16579.843927] Lustre: tst250fs-OST0001: new disk, initializing [16579.847430] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [16579.853696] Lustre: 377387:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16579.961509] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [16586.469585] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_default_debug -1 all [16587.848643] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [16587.860471] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [16587.966254] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [16593.032761] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [16595.805433] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [16595.810727] Lustre: Skipped 2 previous similar messages [16623.592143] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16623.604626] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16623.649241] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [16624.111156] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16624.136370] Lustre: Skipped 1 previous similar message [16624.155473] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [16624.158030] Lustre: Skipped 2 previous similar messages [16629.372766] Lustre: server umount tst250fs-MDT0000 complete [16633.820926] LustreError: 374875:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785404185 with bad export cookie 13096469969075521924 [16633.828655] LustreError: MGC192.168.201.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16633.840251] LustreError: 374875:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [16634.222703] Lustre: server umount tst250fs-MDT0001 complete [16649.173641] Lustre: server umount tst250fs-OST0000 complete [16649.584849] Lustre: 373106:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785404185/real 1785404185] req@ffffa0bbbf570700 x1872131787920896/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785404201 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16649.642859] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16649.663195] Lustre: Skipped 1 previous similar message [16653.540101] Lustre: 373104:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785404189/real 1785404189] req@ffffa0ba879d0a80 x1872131787921152/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785404205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16654.798590] Lustre: server umount tst250fs-OST0001 complete [16677.972525] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [16688.569540] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [16730.392199] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 05:38:00 (1785404280) [16732.627912] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [16759.547882] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing set_hostid [16782.645384] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing load_modules_local [16825.787157] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16438 sec ========= 05:39:36 (1785404376) [16828.579649] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:39:38 (1785404378) === [16833.661490] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:39:43 (1785404383) === [16892.319743] Lustre: DEBUG MARKER: oleg120-server.virtnet: executing unload_modules_local [16896.006202] Key type lgssc unregistered [16896.515591] LNet: 386563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16896.529040] LNetError: 386563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16896.556911] LNet: Removed LNI 192.168.201.120@tcp [16898.130189] Key type .llcrypt unregistered [16898.133340] Key type ._llcrypt unregistered