[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 515294201 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003408] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007957] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011050] Yama: becoming mindful. [ 0.012048] SELinux: Initializing. [ 0.014042] *** VALIDATE selinux *** [ 0.021762] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026019] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028099] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030061] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031088] *** VALIDATE tmpfs *** [ 0.033379] *** VALIDATE proc *** [ 0.034229] *** VALIDATE cgroup *** [ 0.036008] *** VALIDATE cgroup2 *** [ 0.037275] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040023] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.043954] debug: unmapping init [mem 0xffffffffa3259000-0xffffffffa3260fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045648] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047020] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053292] rcu: Hierarchical SRCU implementation. [ 0.055505] smp: Bringing up secondary CPUs ... [ 0.056600] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.061240] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144232] node 0 deferred pages initialised in 76ms [ 0.149139] devtmpfs: initialized [ 0.151179] x86/mm: Memory block size: 128MB [ 0.155408] gcov: version magic: 0x41383552 [ 0.158351] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160230] pinctrl core: initialized pinctrl subsystem [ 0.161268] [ 0.161889] ************************************************************* [ 0.165014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169021] ** ** [ 0.171011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174012] ** ** [ 0.176012] ** This means that this kernel is built to expose internal ** [ 0.179012] ** IOMMU data structures, which may compromise security on ** [ 0.181010] ** your system. ** [ 0.183011] ** ** [ 0.186011] ** If you see this message and you are not debugging the ** [ 0.189016] ** kernel, report this immediately to your vendor! ** [ 0.191012] ** ** [ 0.194011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.196012] ************************************************************* [ 0.199456] NET: Registered protocol family 16 [ 0.201443] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.204057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.207169] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211027] cpuidle: using governor menu [ 0.212500] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.215697] PCI: Using configuration type 1 for base access [ 0.218127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.227092] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.230108] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.234114] cryptd: max_cpu_qlen set to 1000 [ 0.237217] ACPI: Added _OSI(Module Device) [ 0.239010] ACPI: Added _OSI(Processor Device) [ 0.240010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.242011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.247521] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.253399] ACPI: Interpreter enabled [ 0.255051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.256006] ACPI: Using IOAPIC for interrupt routing [ 0.257078] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260396] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.270962] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.273045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.275019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.288526] acpiphp: Slot [2] registered [ 0.290177] acpiphp: Slot [5] registered [ 0.292103] acpiphp: Slot [6] registered [ 0.294119] acpiphp: Slot [7] registered [ 0.296025] acpiphp: Slot [8] registered [ 0.298131] acpiphp: Slot [9] registered [ 0.301130] acpiphp: Slot [10] registered [ 0.302000] acpiphp: Slot [3] registered [ 0.302146] acpiphp: Slot [4] registered [ 0.304240] acpiphp: Slot [11] registered [ 0.306303] acpiphp: Slot [12] registered [ 0.308173] acpiphp: Slot [13] registered [ 0.309124] acpiphp: Slot [14] registered [ 0.311168] acpiphp: Slot [15] registered [ 0.313176] acpiphp: Slot [16] registered [ 0.316171] acpiphp: Slot [17] registered [ 0.318193] acpiphp: Slot [18] registered [ 0.320144] acpiphp: Slot [19] registered [ 0.321142] acpiphp: Slot [20] registered [ 0.324160] acpiphp: Slot [21] registered [ 0.325140] acpiphp: Slot [22] registered [ 0.327153] acpiphp: Slot [23] registered [ 0.329158] acpiphp: Slot [24] registered [ 0.331188] acpiphp: Slot [25] registered [ 0.332115] acpiphp: Slot [26] registered [ 0.334157] acpiphp: Slot [27] registered [ 0.336120] acpiphp: Slot [28] registered [ 0.337191] acpiphp: Slot [29] registered [ 0.338158] acpiphp: Slot [30] registered [ 0.340151] acpiphp: Slot [31] registered [ 0.342120] PCI host bridge to bus 0000:00 [ 0.344029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.346030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.348040] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.351034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.353108] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.357036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.358242] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.362070] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.365358] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.376887] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.380000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.384036] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.387024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.389022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.392751] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.395978] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.401065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.404520] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.411023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.427023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.435022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.442788] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.452000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.461024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.487024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.499551] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.508021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.516022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.541154] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.554000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.565018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.573026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.594026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.606304] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.613018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.626012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.649025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.660900] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.674019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.688021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.707018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.719962] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.732020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.739023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.756017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.766334] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.769434] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.775336] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.777425] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.780286] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.786611] iommu: Default domain type: Passthrough [ 0.788643] SCSI subsystem initialized [ 0.791319] ACPI: bus type USB registered [ 0.793162] usbcore: registered new interface driver usbfs [ 0.796151] usbcore: registered new interface driver hub [ 0.799500] usbcore: registered new device driver usb [ 0.802235] pps_core: LinuxPPS API ver. 1 registered [ 0.804015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.807110] PTP clock support registered [ 0.810075] EDAC MC: Ver: 3.0.0 [ 0.812225] PCI: Using ACPI for IRQ routing [ 0.815684] NetLabel: Initializing [ 0.816000] NetLabel: domain hash size = 128 [ 0.816000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.819109] NetLabel: unlabeled traffic allowed by default [ 0.822102] vgaarb: loaded [ 0.823298] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.826025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.833614] clocksource: Switched to clocksource kvm-clock [ 0.949141] VFS: Disk quotas dquot_6.6.0 [ 0.951214] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.954478] *** VALIDATE ramfs *** [ 0.955945] *** VALIDATE hugetlbfs *** [ 0.957750] pnp: PnP ACPI init [ 0.960475] pnp: PnP ACPI: found 6 devices [ 0.983687] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.988704] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.991390] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.997382] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.001419] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.007921] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.011092] NET: Registered protocol family 2 [ 1.015135] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.021149] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.025207] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.030445] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.034456] TCP: Hash tables configured (established 65536 bind 65536) [ 1.037483] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.042528] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.046141] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.049713] NET: Registered protocol family 1 [ 1.053762] RPC: Registered named UNIX socket transport module. [ 1.056255] RPC: Registered udp transport module. [ 1.058043] RPC: Registered tcp transport module. [ 1.060079] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.062753] NET: Registered protocol family 44 [ 1.065238] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.068103] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.071056] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.073727] PCI: CLS 0 bytes, default 64 [ 1.075466] Unpacking initramfs... [ 2.619134] debug: unmapping init [mem 0xffff8b82fcc54000-0xffff8b82fffbffff] [ 2.625784] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.628269] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.633081] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.160424] Initialise system trusted keyrings [ 3.162527] Key type blacklist registered [ 3.164744] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.179429] zbud: loaded [ 3.183343] *** VALIDATE nfs *** [ 3.185398] *** VALIDATE nfs4 *** [ 3.186958] pstore: using deflate compression [ 3.191710] Platform Keyring initialized [ 3.316799] NET: Registered protocol family 38 [ 3.318795] Key type asymmetric registered [ 3.320486] Asymmetric key parser 'x509' registered [ 3.322569] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.325842] io scheduler mq-deadline registered [ 3.328054] io scheduler kyber registered [ 3.330117] io scheduler bfq registered [ 3.332397] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.335533] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.338738] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.342389] ACPI: Power Button [PWRF] [ 3.348074] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.356687] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.373734] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.381182] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.402349] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.430536] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.463642] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.469717] Non-volatile memory driver v1.3 [ 3.471956] Linux agpgart interface v0.103 [ 3.509148] virtio_blk virtio1: [vda] 150616 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.515468] vda: detected capacity change from 0 to 77115392 [ 3.539184] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.544053] vdb: detected capacity change from 0 to 1073741824 [ 3.561313] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.565508] vdc: detected capacity change from 0 to 2621440000 [ 3.586794] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.591570] vdd: detected capacity change from 0 to 2621440000 [ 3.610184] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.614785] vde: detected capacity change from 0 to 4294967296 [ 3.641669] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.645234] vdf: detected capacity change from 0 to 4294967296 [ 3.652239] libphy: Fixed MDIO Bus: probed [ 3.657574] usbcore: registered new interface driver usbserial_generic [ 3.660268] usbserial: USB Serial support registered for generic [ 3.662891] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.668303] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.670732] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.674214] mousedev: PS/2 mouse device common for all mice [ 3.678224] rtc_cmos 00:05: RTC can wake from S4 [ 3.682068] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.683226] rtc_cmos 00:05: registered as rtc0 [ 3.689957] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.691802] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.694779] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.694856] intel_pstate: CPU model not supported [ 3.706972] hid: raw HID events driver (C) Jiri Kosina [ 3.709961] usbcore: registered new interface driver usbhid [ 3.712872] usbhid: USB HID core driver [ 3.715199] drop_monitor: Initializing network drop monitor service [ 3.719562] Initializing XFRM netlink socket [ 3.721895] NET: Registered protocol family 10 [ 3.725233] Segment Routing with IPv6 [ 3.727032] NET: Registered protocol family 17 [ 3.729054] mpls_gso: MPLS GSO support [ 3.734280] RAS: Correctable Errors collector initialized. [ 3.736243] AVX version of gcm_enc/dec engaged. [ 3.738717] AES CTR mode by8 optimization enabled [ 3.827863] sched_clock: Marking stable (3827772524, 0)->(4785900535, -958128011) [ 3.832437] registered taskstats version 1 [ 3.835206] Loading compiled-in X.509 certificates [ 3.838121] zswap: loaded using pool lzo/zbud [ 3.870286] Key type big_key registered [ 3.883831] Key type encrypted registered [ 3.885871] ima: No TPM chip found, activating TPM-bypass! [ 3.888391] ima: Allocated hash algorithm: sha1 [ 3.890711] ima: No architecture policies found [ 3.893177] evm: Initialising EVM extended attributes: [ 3.895660] evm: security.selinux [ 3.897789] evm: security.ima [ 3.899099] evm: security.capability [ 3.901215] evm: HMAC attrs: 0x1 [ 3.904705] rtc_cmos 00:05: setting system clock to 2026-10-02 13:07:40 UTC (1790946460) [ 3.912647] debug: unmapping init [mem 0xffffffffa4203000-0xffffffffa43fffff] [ 3.916051] debug: unmapping init [mem 0xffffffffa2f82000-0xffffffffa3258fff] [ 3.925155] Write protecting the kernel read-only data: 28672k [ 3.930120] debug: unmapping init [mem 0xffffffffa1603000-0xffffffffa17fffff] [ 3.933449] debug: unmapping init [mem 0xffffffffa1f14000-0xffffffffa1ffffff] [ 3.970339] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.979666] systemd[1]: Detected virtualization kvm. [ 3.982065] systemd[1]: Detected architecture x86-64. [ 3.984240] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.013553] systemd[1]: No hostname configured. [ 4.016168] systemd[1]: Set hostname to . [ 4.018979] random: systemd: uninitialized urandom read (16 bytes read) [ 4.021984] systemd[1]: Initializing machine ID from random generator. [ 4.062922] random: ln: uninitialized urandom read (6 bytes read) [ 4.192275] random: systemd: uninitialized urandom read (16 bytes read) [ 4.195459] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.202133] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.211795] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.887189] device-mapper: uevent: version 1.0.3 [ 4.890631] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.641765] virtio_net virtio0 ens2: renamed from eth0 [ 5.659211] random: fast init done [ 5.752714] scsi host0: ata_piix [ 5.786708] scsi host1: ata_piix [ 5.798173] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.800429] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.409865] random: crng init done [ 10.412011] random: 7 urandom warning(s) missed due to ratelimiting [ 10.679882] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.272277] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.192827] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.877985] SELinux: Disabled at runtime. [ 16.017446] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.056385] systemd[1]: Detected virtualization kvm. [ 16.060433] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.828256] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.844401] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.871952] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.882799] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.889310] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.903347] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.908933] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 18.071734] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 19.222228] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.404401] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.486343] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.948292] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.970050] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 25.307162] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 26.252898] NFS: Registering the id_resolver key type [ 26.256752] Key type id_resolver registered [ 26.259626] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg636-server login: [ 78.150883] libcfs: loading out-of-tree module taints kernel. [ 78.212930] Key type ._llcrypt registered [ 78.214873] Key type .llcrypt registered [ 78.321097] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 94.895476] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 96.376471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 96.391185] alg: No test for adler32 (adler32-zlib) [ 97.797365] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 98.514511] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 100.273911] Key type lgssc registered [ 101.836294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.373427] hrtimer: interrupt took 4414258 ns [ 117.059827] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 162.600839] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 176.573231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 176.599565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.822596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 177.866192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 177.973353] Lustre: lustre-MDT0000: new disk, initializing [ 178.027628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.044327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 181.811364] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.093664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.221400] Lustre: 6539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 195.265692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 195.275114] Lustre: Skipped 1 previous similar message [ 195.366792] Lustre: lustre-MDT0001: new disk, initializing [ 195.439635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 195.475062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 195.489751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 199.359629] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.828313] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 212.114199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.314764] Lustre: lustre-OST0000: new disk, initializing [ 212.319490] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 212.321049] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 212.321510] Lustre: 8477:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.392690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.359178] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.627448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.639572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 217.674797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 230.663950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 230.821498] Lustre: lustre-OST0001: new disk, initializing [ 230.824892] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 230.830282] Lustre: 9547:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 230.888792] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.102372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 237.115610] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 237.211175] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 237.365099] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.176238] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 259.275464] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 265.201081] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing check_logdir /tmp/testlogs/ [ 270.301550] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing yml_node [ 274.466537] Lustre: DEBUG MARKER: Client: 2.17.59.3 [ 276.756715] Lustre: DEBUG MARKER: MDS: 2.17.59.3 [ 279.020950] Lustre: DEBUG MARKER: OSS: 2.17.59.3 [ 280.530488] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 2 09:12:14 EDT 2026 [ 297.036768] Lustre: DEBUG MARKER: only running tests: 121-999 [ 298.865748] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 300.386777] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 318.432937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 318.441519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.459296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.458449] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.465871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.473325] Lustre: Skipped 2 previous similar messages [ 319.478535] Lustre: Skipped 2 previous similar messages [ 324.380791] Lustre: server umount lustre-MDT0000 complete [ 328.145887] LustreError: 6531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790946784 with bad export cookie 15167764276368763014 [ 328.148418] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 328.162195] LustreError: 6531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 329.700346] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.701714] LustreError: 6547:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.708195] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.708206] Lustre: Skipped 1 previous similar message [ 329.711678] Lustre: Skipped 1 previous similar message [ 329.749984] LustreError: 6547:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 333.729140] LustreError: 6546:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.734878] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 333.757153] LustreError: 6546:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 333.788346] Lustre: Skipped 2 previous similar messages [ 334.599652] Lustre: server umount lustre-MDT0001 complete [ 338.184198] Lustre: server umount lustre-OST0000 complete [ 342.054297] Lustre: server umount lustre-OST0001 complete [ 347.770729] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 356.327349] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 387.752066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.975661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 387.998331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 388.062313] Lustre: lustre-MDT0000: new disk, initializing [ 388.130071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.148361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 392.749107] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 402.255657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.403904] Lustre: 15148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 402.495309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 402.504945] Lustre: Skipped 1 previous similar message [ 402.590701] Lustre: lustre-MDT0001: new disk, initializing [ 402.679435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 402.712283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 402.720072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 406.876854] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 415.489581] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 420.518708] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 426.461741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 426.744084] Lustre: lustre-OST0000: new disk, initializing [ 426.760414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 426.770332] Lustre: 17118:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 426.858094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 427.928037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 427.948092] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 428.042674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 432.913682] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 442.868973] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 447.945786] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 448.174907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 452.464956] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 452.677603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 458.722770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 458.726761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 458.758430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 468.960774] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 469.161768] Lustre: server umount lustre-OST0000 complete [ 474.087735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.104744] Lustre: Skipped 2 previous similar messages [ 478.852256] Lustre: server umount lustre-MDT0000 complete [ 479.201592] LustreError: 15949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.230587] LustreError: 15949:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 482.646533] LustreError: 17117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790946939 with bad export cookie 15167764276368764540 [ 482.663452] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 482.669283] LustreError: 17117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 482.986692] Lustre: server umount lustre-MDT0001 complete [ 489.737614] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:15:43 (1790946943) [ 508.290143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.800417] LustreError: 19992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.921642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.251312] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 514.019064] LustreError: 19993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.087661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.133391] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 533.783354] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 538.539817] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 540.337296] Lustre: Failing over lustre-MDT0000 [ 540.603439] Lustre: server umount lustre-MDT0000 complete [ 542.185501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 542.199932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.226112] LustreError: 19993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.252663] LustreError: 19993:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 558.440037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.521786] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 562.484885] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 564.202164] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 564.212855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 564.247071] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 571.960930] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 579.553978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.558565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 579.578063] Lustre: Skipped 2 previous similar messages [ 579.587285] Lustre: Skipped 8 previous similar messages [ 590.304144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 590.490951] Lustre: server umount lustre-MDT0000 complete [ 591.850464] LustreError: 19989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 591.868920] LustreError: 19989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 593.980882] LustreError: 19972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947050 with bad export cookie 15167764276368765961 [ 593.999260] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.275358] Lustre: server umount lustre-MDT0001 complete [ 601.494917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.922468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.926778] Lustre: Skipped 2 previous similar messages [ 605.705775] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 613.717288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 618.062685] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 624.754583] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 630.378230] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 632.495810] Lustre: Failing over lustre-MDT0000 [ 632.775304] Lustre: server umount lustre-MDT0000 complete [ 634.337493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 634.344777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.374638] LustreError: 23327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.390465] LustreError: 23327:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 651.944775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 652.183482] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 652.534596] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 656.596673] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 657.890870] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 657.910821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 657.920563] Lustre: Skipped 1 previous similar message [ 657.950665] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 665.386594] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 672.232219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 672.243557] Lustre: Skipped 5 previous similar messages [ 675.271976] Lustre: server umount lustre-MDT0000 complete [ 679.063044] LustreError: 23307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947135 with bad export cookie 15167764276368766899 [ 679.073431] LustreError: 23307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 679.081915] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 679.346635] Lustre: server umount lustre-MDT0001 complete [ 687.825111] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:19:02 (1790947142) [ 702.592359] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 711.693056] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 753.530753] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 764.276304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 764.563517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 764.594541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 764.682120] Lustre: lustre-MDT0000: new disk, initializing [ 764.799274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 764.805244] Lustre: Skipped 2 previous similar messages [ 764.834565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 769.294924] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 779.907207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 780.011778] Lustre: 29702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 780.243979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 784.279086] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 792.705257] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 797.646560] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 804.081098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 804.355375] Lustre: 31671:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 805.837847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 805.908654] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 805.974110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 806.002788] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 811.752145] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 821.102837] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 826.422921] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 846.821049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 846.825354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 846.830309] Lustre: Skipped 3 previous similar messages [ 846.831417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 846.844741] Lustre: Skipped 1 previous similar message [ 850.832851] Lustre: server umount lustre-OST0000 complete [ 857.059352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 857.066830] LustreError: Skipped 1 previous similar message [ 860.954620] Lustre: server umount lustre-MDT0000 complete [ 862.196081] LustreError: 29709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 862.222854] LustreError: 29709:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 864.861666] LustreError: 29693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947321 with bad export cookie 15167764276368767305 [ 864.862696] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 864.884197] LustreError: 29693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 865.504240] Lustre: server umount lustre-MDT0001 complete [ 878.785029] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 881.546782] Key type lgssc unregistered [ 881.908882] LNet: 33648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.921425] LNetError: 33648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.941220] LNet: Removed LNI 192.168.206.136@tcp [ 882.907156] Key type .llcrypt unregistered [ 882.908972] Key type ._llcrypt unregistered [ 906.527865] Key type ._llcrypt registered [ 906.535875] Key type .llcrypt registered [ 906.622678] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 907.625600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 907.656294] alg: No test for adler32 (adler32-zlib) [ 908.693297] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 908.894548] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 910.568193] Key type lgssc registered [ 911.667068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 918.447603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 918.478243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.905392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.801894] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 930.282314] LustreError: 35079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 930.297513] LustreError: 35079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 930.679784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.085484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 934.792183] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 941.104352] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 946.328819] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 952.183780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 952.654717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.702970] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 958.584119] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 964.925646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 965.339491] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 977.022068] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:23:51 (1790947431) [ 978.548959] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 979.955159] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 985.324727] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:23:59 (1790947439) [ 993.063925] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:24:07 (1790947447) [ 994.038162] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1001.197763] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1001.203355] Lustre: Skipped 2 previous similar messages [ 1002.931375] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:24:17 (1790947457) [ 1006.180547] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1006.186760] Lustre: Skipped 3 previous similar messages [ 1021.388283] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1021.401477] Lustre: Skipped 6 previous similar messages [ 1024.492488] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:24:38 (1790947478) [ 1027.166457] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1030.386181] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1038.428338] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:24:52 (1790947492) [ 1039.990993] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1039.995763] Lustre: Skipped 3 previous similar messages [ 1050.441229] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:25:04 (1790947504) [ 1062.373476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1062.375514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1062.399466] Lustre: Skipped 1 previous similar message [ 1062.410093] Lustre: Skipped 1 previous similar message [ 1067.494829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1067.500759] Lustre: Skipped 1 previous similar message [ 1072.615966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1072.628083] Lustre: Skipped 1 previous similar message [ 1076.193197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1076.308428] Lustre: server umount lustre-OST0000 complete [ 1079.778921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1079.788767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.802332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1082.850455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.545532] Lustre: server umount lustre-MDT0000 complete [ 1087.973753] LustreError: 35811:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1087.992928] LustreError: 35811:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1089.260229] LustreError: 35057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947545 with bad export cookie 1683088383153012928 [ 1089.263870] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1089.291119] LustreError: 35057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1089.567628] Lustre: server umount lustre-MDT0001 complete [ 1100.554887] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 1103.047568] Key type lgssc unregistered [ 1103.276251] LNet: 42930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.284409] LNetError: 42930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1104.364689] LNet: Removed LNI 192.168.206.136@tcp [ 1105.229196] Key type .llcrypt unregistered [ 1105.234834] Key type ._llcrypt unregistered [ 1122.134388] Key type ._llcrypt registered [ 1122.137980] Key type .llcrypt registered [ 1122.216230] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 1133.803336] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 1134.543359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1134.564694] alg: No test for adler32 (adler32-zlib) [ 1135.514707] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 1135.699716] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 1137.330427] Key type lgssc registered [ 1138.191658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1162.138461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1162.158485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1163.388695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1163.418975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1163.477764] Lustre: lustre-MDT0000: new disk, initializing [ 1163.573222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1163.591690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1167.751416] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1175.778525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1175.965874] Lustre: 46077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1176.037831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1176.047958] Lustre: Skipped 1 previous similar message [ 1176.131309] Lustre: lustre-MDT0001: new disk, initializing [ 1176.248665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1176.308473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1176.333692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1181.108846] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1191.988744] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1198.268869] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1205.624555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1206.033451] Lustre: lustre-OST0000: new disk, initializing [ 1206.037688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1206.046096] Lustre: 48045:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1206.178215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1212.078887] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1212.961153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1212.990880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1213.180054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1221.336429] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1225.821452] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1225.963415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1230.423685] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1230.580734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1233.895822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1233.896617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.902329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1233.902537] LustreError: Skipped 1 previous similar message [ 1233.908136] Lustre: Skipped 1 previous similar message [ 1238.731619] Lustre: server umount lustre-OST0000 complete [ 1243.106293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1243.110143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.121837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1243.127990] Lustre: Skipped 1 previous similar message [ 1244.130239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1244.134125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.703905] Lustre: server umount lustre-MDT0000 complete [ 1250.736845] LustreError: 46067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947707 with bad export cookie 5989953170732004439 [ 1250.739193] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1250.747328] LustreError: 46067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1251.024986] Lustre: server umount lustre-MDT0001 complete [ 1258.712585] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:28:33 (1790947713) [ 1264.520397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1265.013092] LustreError: 50504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1265.032729] LustreError: 50504:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1265.081468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.309736] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1270.244627] LustreError: 50505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.363702] LustreError: 50504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.980088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.243796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1280.923797] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1287.213151] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1291.794932] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1297.015445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1297.265517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1301.822509] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1307.709328] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1315.508954] LustreError: 53033:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1316.149834] Lustre: Modifying parameter general.timeout=1 in log params [ 1317.417375] Lustre: Modifying parameter general.timeout=3 in log params [ 1317.421726] Lustre: Skipped 1 previous similar message [ 1319.448237] Lustre: Modifying parameter general.timeout=6 in log params [ 1319.452507] Lustre: Skipped 2 previous similar messages [ 1323.527236] Lustre: Modifying parameter general.timeout=11 in log params [ 1323.531372] Lustre: Skipped 4 previous similar messages [ 1332.134552] Lustre: Modifying parameter general.timeout=23 in log params [ 1332.139699] Lustre: Skipped 11 previous similar messages [ 1348.734228] Lustre: Modifying parameter general.timeout=45 in log params [ 1348.742593] Lustre: Skipped 21 previous similar messages [ 1381.078457] Lustre: Modifying parameter general.timeout=80 in log params [ 1381.081528] Lustre: Skipped 34 previous similar messages [ 1429.663422] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:31:24 (1790947884) [ 1432.464943] Lustre: 59965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1434.223598] Lustre: 60073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1437.709299] Lustre: 60272:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1437.720501] Lustre: 60272:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1445.987505] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1445.998655] Lustre: Skipped 60 previous similar messages [ 1463.481657] Lustre: 61839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1463.489301] Lustre: 61839:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1463.506600] LustreError: 61839:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1467.283077] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:32:01 (1790947921) [ 1481.192931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.198412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.207672] Lustre: Skipped 2 previous similar messages [ 1481.217167] Lustre: Skipped 3 previous similar messages [ 1483.963419] Lustre: server umount lustre-MDT0000 complete [ 1486.325098] LustreError: 52994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1486.351040] LustreError: 52994:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1487.575063] LustreError: 57833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947944 with bad export cookie 5989953170732005321 [ 1487.576910] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1487.597952] LustreError: 57833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1487.866924] Lustre: server umount lustre-MDT0001 complete [ 1500.611261] Lustre: server umount lustre-OST0000 complete [ 1520.855689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1521.082244] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1521.364713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1525.523060] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1533.396651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.518033] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1533.561494] Lustre: 64367:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1538.440318] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1544.694774] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1549.060476] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1553.931640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1554.136539] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1554.413582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1554.418662] Lustre: Skipped 1 previous similar message [ 1559.753413] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1565.806987] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1567.205559] Lustre: 66931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1573.858696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1573.875609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1573.900200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1579.840535] Lustre: server umount lustre-OST0000 complete [ 1584.610352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1584.614802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1584.624223] Lustre: Skipped 2 previous similar messages [ 1584.629187] Lustre: Skipped 4 previous similar messages [ 1589.540778] Lustre: server umount lustre-MDT0000 complete [ 1589.733782] LustreError: 64378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1589.753703] LustreError: 64378:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1593.064854] LustreError: 64360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948049 with bad export cookie 5989953170732008240 [ 1593.069067] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1593.079234] LustreError: 64360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1593.362871] Lustre: server umount lustre-MDT0001 complete [ 1602.246922] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 1604.948676] Key type lgssc unregistered [ 1605.214041] LNet: 68141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.224259] LNetError: 68141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.246252] LNet: Removed LNI 192.168.206.136@tcp [ 1605.968708] Key type .llcrypt unregistered [ 1605.971118] Key type ._llcrypt unregistered [ 1626.653423] Key type ._llcrypt registered [ 1626.655639] Key type .llcrypt registered [ 1626.738359] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 1627.508342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1627.525288] alg: No test for adler32 (adler32-zlib) [ 1628.547199] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 1628.755553] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 1630.425363] Key type lgssc registered [ 1631.374432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1637.809738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1637.836316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1639.316279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.721238] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1648.610395] LustreError: 69562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1648.645500] LustreError: 69562:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1651.438715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.212762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1656.062945] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1661.714550] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1666.765894] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1672.407503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1673.084503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1680.264922] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1687.711709] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1694.652446] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1702.883073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1702.889251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.897521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1704.932847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.933339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1704.966265] Lustre: Skipped 1 previous similar message [ 1708.678640] Lustre: server umount lustre-MDT0000 complete [ 1710.052474] LustreError: 69562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1710.073651] LustreError: 69562:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1711.942645] LustreError: 69544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948168 with bad export cookie 16401500807283000914 [ 1711.945964] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1711.951656] LustreError: 69544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1712.218393] Lustre: server umount lustre-MDT0001 complete [ 1726.179751] Lustre: server umount lustre-OST0000 complete [ 1750.328862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1750.689896] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1750.747425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1751.088820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.146230] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1764.014350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1764.228453] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1764.281559] Lustre: 74432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1764.851847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1769.929857] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1776.799576] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1781.713877] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1787.530547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1787.778184] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1788.102956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.817319] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1800.089204] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1801.463468] Lustre: 77007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1803.069668] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1807.844102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1807.879108] Lustre: Skipped 1 previous similar message [ 1807.893507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1812.456856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1812.468854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1812.486236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1813.814883] Lustre: server umount lustre-OST0000 complete [ 1818.081669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.098633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1818.103787] Lustre: Skipped 1 previous similar message [ 1824.048087] Lustre: server umount lustre-MDT0000 complete [ 1827.715977] LustreError: 77009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948284 with bad export cookie 16401500807283001747 [ 1827.723303] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.728827] LustreError: 77009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1828.088747] Lustre: server umount lustre-MDT0001 complete [ 1838.209936] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 1840.731678] Key type lgssc unregistered [ 1840.993904] LNet: 78216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.002966] LNetError: 78216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.027848] LNet: Removed LNI 192.168.206.136@tcp [ 1841.779303] Key type .llcrypt unregistered [ 1841.784650] Key type ._llcrypt unregistered [ 1864.672533] Key type ._llcrypt registered [ 1864.682884] Key type .llcrypt registered [ 1864.795090] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 1865.644537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1865.728874] alg: No test for adler32 (adler32-zlib) [ 1866.847595] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 1867.075954] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 1868.736332] Key type lgssc registered [ 1869.885589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1876.547647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1876.575635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.942387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.408910] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1888.226160] LustreError: 79643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1888.245136] LustreError: 79643:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1889.452212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.097331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1894.120604] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1900.887734] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1906.089670] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1912.696821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1913.320940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1919.259906] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 1927.200599] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1932.963914] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1933.839956] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1935.702252] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1935.706989] Lustre: Skipped 1 previous similar message [ 1938.459501] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1938.470518] Lustre: Skipped 2 previous similar messages [ 1942.836328] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1942.840680] Lustre: Skipped 4 previous similar messages [ 1951.651276] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1951.660673] Lustre: Skipped 9 previous similar messages [ 1967.978815] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1967.982552] Lustre: Skipped 16 previous similar messages [ 2000.507990] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2000.512127] Lustre: Skipped 37 previous similar messages [ 2029.619703] Lustre: 87742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2032.755585] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:41:26 (1790948486) [ 2034.572751] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2036.824245] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:41:30 (1790948490) [ 2040.296638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2040.297643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.337048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2040.347528] Lustre: Skipped 1 previous similar message [ 2045.410075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2050.549705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2050.557338] Lustre: Skipped 2 previous similar messages [ 2054.115289] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2054.241855] Lustre: server umount lustre-OST0000 complete [ 2058.721103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2058.735541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.755460] Lustre: Skipped 1 previous similar message [ 2058.762317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.770866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2065.893958] LustreError: 79643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.919823] LustreError: 79643:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2066.449698] Lustre: server umount lustre-MDT0000 complete [ 2070.457571] LustreError: 81494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948527 with bad export cookie 10748976552025483864 [ 2070.463406] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2070.474768] LustreError: 81494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2070.994392] Lustre: server umount lustre-MDT0001 complete [ 2080.269986] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2083.190612] Key type lgssc unregistered [ 2083.465345] LNet: 88990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2083.473808] LNetError: 88990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2084.518071] LNet: Removed LNI 192.168.206.136@tcp [ 2085.577077] Key type .llcrypt unregistered [ 2085.584421] Key type ._llcrypt unregistered [ 2091.149764] Key type ._llcrypt registered [ 2091.152207] Key type .llcrypt registered [ 2091.230861] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2096.057811] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules [ 2097.836196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2097.858848] alg: No test for adler32 (adler32-zlib) [ 2099.034251] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 2099.316877] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 2101.064501] Key type lgssc registered [ 2102.449371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2103.753417] LustreError: 90221:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2119.924458] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2124.955564] LustreError: 90221:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2129.460465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2129.496699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.028612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2135.860963] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2141.155488] LustreError: 91233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.184850] LustreError: 91233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2144.990833] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:43:19 (1790948599) [ 2146.277590] LustreError: 91234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.154461] LustreError: 91266:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2149.164596] LustreError: 91266:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2149.182089] LustreError: 91266:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2149.665164] Lustre: server umount lustre-MDT0000 complete [ 2163.744693] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 2166.749619] Key type lgssc unregistered [ 2167.075043] LNet: 92760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.086145] LNetError: 92760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2167.120225] LNet: Removed LNI 192.168.206.136@tcp [ 2168.055081] Key type .llcrypt unregistered [ 2168.057501] Key type ._llcrypt unregistered [ 2184.096775] Key type ._llcrypt registered [ 2184.100715] Key type .llcrypt registered [ 2184.197577] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2185.316618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2185.348226] alg: No test for adler32 (adler32-zlib) [ 2186.387930] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 2186.665465] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 2188.320227] Key type lgssc registered [ 2189.600651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2196.465569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2196.484090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.005857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.975195] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2208.225709] LustreError: 93990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2208.247416] LustreError: 93990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2209.943569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.611580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2214.971870] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2221.939753] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2227.431734] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2234.959364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2235.375748] LustreError: 95819:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.132671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.410686] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2252.002657] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2301.065956] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:45:55 (1790948755) [ 2302.995666] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2304.844642] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:45:59 (1790948759) [ 2309.099522] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.107124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2309.123254] Lustre: Skipped 1 previous similar message [ 2309.131184] Lustre: Skipped 2 previous similar messages [ 2314.211802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2314.224723] Lustre: Skipped 2 previous similar messages [ 2314.618703] Lustre: server umount lustre-MDT0000 complete [ 2318.426474] LustreError: 94732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948775 with bad export cookie 13683450415715611416 [ 2318.436145] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2318.452488] LustreError: 94732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2319.018693] Lustre: server umount lustre-MDT0001 complete [ 2333.446207] Lustre: server umount lustre-OST0000 complete [ 2345.402853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.877292] LustreError: 98267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.086703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.699486] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2351.079694] LustreError: 98268:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.096211] LustreError: 98268:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2356.199432] LustreError: 98267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.654090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.580923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2366.100751] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2373.510854] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2378.024776] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2390.976966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2391.273396] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2391.279152] LustreError: 98257:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2391.306142] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2391.311898] LustreError: 100351:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2391.317648] LustreError: 100351:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2391.323816] LustreError: 100351:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2391.418142] Lustre: server umount lustre-OST0000 complete [ 2391.420388] LustreError: 100351:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2398.945083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2399.182789] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2399.189637] LustreError: 98256:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2399.213811] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2399.223380] LustreError: 100817:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2399.235842] LustreError: 100817:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2399.240962] LustreError: 100817:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2399.326493] Lustre: server umount lustre-OST0000 complete [ 2399.332186] LustreError: 100817:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2407.779423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2407.998944] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2408.014028] Lustre: Found index 0 for lustre-OST0000, updating log [ 2408.024237] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2408.085364] Lustre: lustre-OST0000: new disk, initializing [ 2408.089231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2408.093320] Lustre: 101347:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2408.427379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.975603] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2417.172503] Lustre: Failing over lustre-OST0000 [ 2417.319371] Lustre: server umount lustre-OST0000 complete [ 2421.729497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2421.742586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.757540] Lustre: Skipped 1 previous similar message [ 2421.766563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.825334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.838831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.988080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.996807] Lustre: Skipped 2 previous similar messages [ 2431.094987] Lustre: server umount lustre-MDT0000 complete [ 2434.088721] LustreError: 100370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948890 with bad export cookie 13683450415715614167 [ 2434.099875] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2434.106859] LustreError: 100370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2434.501601] Lustre: server umount lustre-MDT0001 complete [ 2444.106457] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:48:17 (1790948897) [ 2460.430769] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2469.838494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2470.353813] LustreError: 103691:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.485727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2474.867509] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2483.190337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.950899] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2492.323072] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2498.705845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2499.399673] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2505.306293] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2507.573381] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2507.580380] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2507.589985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2512.885046] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2512.910364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2513.368811] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2513.474922] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2513.595301] Lustre: lustre-OST0001: new disk, initializing [ 2513.601478] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2513.611725] Lustre: 106009:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2513.911049] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 2513.926956] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2513.946101] Lustre: Skipped 2 previous similar messages [ 2514.928733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2514.936188] LustreError: Skipped 1 previous similar message [ 2514.957470] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2514.963964] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2514.969778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2517.775204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2517.791479] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2517.879313] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2520.486640] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2527.703439] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2533.541539] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2538.440406] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.36@tcp (stopping) [ 2538.466619] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.493523] Lustre: Skipped 1 previous similar message [ 2552.288188] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2552.555303] Lustre: server umount lustre-MDT0001 complete [ 2553.743467] LustreError: 106490:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.782209] LustreError: 106490:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2562.542195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2562.692119] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2562.697240] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2562.754788] Lustre: 103678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2564.286624] LustreError: 107558:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2564.294640] LustreError: 107558:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2564.299085] LustreError: 103683:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2564.341541] LustreError: 107564:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.136@tcp: cfg command failed: rc = -17 [ 2564.350697] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2564.350697] [ 2567.813959] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2568.684441] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2577.738974] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:50:32 (1790949032) [ 2594.336523] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2612.056935] Lustre: 109624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2633.460461] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2636.923531] Lustre: 110759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2716.646050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2716.662984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2716.688794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2716.703902] Lustre: Skipped 11 previous similar messages [ 2721.597211] Lustre: server umount lustre-MDT0000 complete [ 2722.279771] LustreError: 106490:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.312385] LustreError: 106490:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 2725.680055] LustreError: 103671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949182 with bad export cookie 13683450415715615098 [ 2725.686886] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2725.695855] LustreError: 103671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2726.025695] Lustre: server umount lustre-MDT0001 complete [ 2740.154172] Lustre: server umount lustre-OST0000 complete [ 2742.496100] Lustre: 93292:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790949183/real 1790949183] req@ffff8b82508bfb80 x1877945764631168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790949199 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2742.525371] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.543498] Lustre: Skipped 3 previous similar messages [ 2744.520328] Lustre: server umount lustre-OST0001 complete [ 2750.337735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2760.131663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2768.855412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2780.019426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2799.956742] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 2811.130240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2811.151183] Lustre: lustre-MDT0000: reset Object Index mappings [ 2811.520509] LustreError: 114707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2811.625911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.637250] Lustre: Skipped 1 previous similar message [ 2817.922054] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2828.861832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2828.879601] Lustre: lustre-MDT0001: reset Object Index mappings [ 2835.238401] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2838.530907] Lustre: 115855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2847.173271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2854.397052] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 2855.608975] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2856.443048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 2864.102545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2869.760757] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2869.762545] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 2871.198883] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 2878.240883] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2881.668897] Lustre: 117733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2925.823752] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:56:19 (1790949379) [ 2952.161697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2952.167100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2952.175710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2952.179163] Lustre: Skipped 4 previous similar messages [ 2958.650926] Lustre: server umount lustre-MDT0000 complete [ 2961.896024] LustreError: 114704:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.920158] LustreError: 114704:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 2962.153650] LustreError: 114687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949418 with bad export cookie 13683450415715662173 [ 2962.158520] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2962.175887] LustreError: 114687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2968.690714] Lustre: server umount lustre-MDT0001 complete [ 2979.327392] Lustre: server umount lustre-OST0000 complete [ 2983.225904] LustreError: 118694:0:(ldlm_resource.c:1227:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0xb8:0x0].0x0 (ffff8b824d97a600) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2989.607387] Lustre: server umount lustre-OST0001 complete [ 2996.155038] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 3005.946789] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 3040.679511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.901864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3040.925373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3040.985348] Lustre: lustre-MDT0000: new disk, initializing [ 3041.052159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.060929] Lustre: Skipped 3 previous similar messages [ 3041.079358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3045.752272] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3052.271551] Lustre: server umount lustre-MDT0000 complete [ 3062.172943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.581629] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3062.591286] Lustre: 122003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3062.611613] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3066.848919] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3072.051105] Lustre: server umount lustre-MDT0000 complete [ 3080.638659] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:58:55 (1790949535) [ 3082.128566] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3083.706180] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:58:58 (1790949538) [ 3100.541260] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 3109.747503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.373402] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3122.545640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.722961] Lustre: 124049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3122.798660] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3122.802026] Lustre: Skipped 1 previous similar message [ 3122.881898] Lustre: lustre-MDT0001: new disk, initializing [ 3123.004184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3123.013143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3127.527879] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3132.322887] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3138.925141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3139.202189] Lustre: lustre-OST0000: new disk, initializing [ 3139.209907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3139.218748] Lustre: 125628:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3140.435795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3140.449024] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3140.562902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3144.791884] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3154.793678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3154.960506] Lustre: lustre-OST0001: new disk, initializing [ 3154.969215] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3154.983619] Lustre: 126498:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3156.635465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3156.644827] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3156.763922] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3162.732792] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3172.695731] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3176.401635] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3548.724345] Lustre: 132927:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3548.733644] Lustre: 132927:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3548.754268] Lustre: 132927:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3548.765219] Lustre: 132927:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3548.784361] Lustre: 132927:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3548.797637] Lustre: 132927:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3825.364389] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:11:18 (1790950278) [ 3843.047980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.053513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.062402] Lustre: Skipped 6 previous similar messages [ 3843.065276] Lustre: Skipped 6 previous similar messages [ 3848.163634] LustreError: 124058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.183335] LustreError: 124058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3848.447957] Lustre: server umount lustre-MDT0000 complete [ 3851.897766] LustreError: 124044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950308 with bad export cookie 13683450415715727735 [ 3851.900315] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3851.907863] LustreError: 124044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3853.283051] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3853.294562] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3853.302973] Lustre: Skipped 3 previous similar messages [ 3853.312245] Lustre: Skipped 4 previous similar messages [ 3858.725467] Lustre: server umount lustre-MDT0001 complete [ 3868.629297] Lustre: server umount lustre-OST0000 complete [ 3878.485980] Lustre: server umount lustre-OST0001 complete [ 3885.009200] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 3895.751241] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 3930.706230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3931.047783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3931.077659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3931.185889] Lustre: lustre-MDT0000: new disk, initializing [ 3931.303890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3931.309740] Lustre: Skipped 5 previous similar messages [ 3931.327251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3936.454066] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3947.977117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3948.082622] Lustre: 135976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3948.142526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3948.147636] Lustre: Skipped 1 previous similar message [ 3948.259548] Lustre: lustre-MDT0001: new disk, initializing [ 3948.441143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3948.457791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3953.244968] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3963.233071] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3968.360163] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3975.089064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3975.392547] Lustre: lustre-OST0000: new disk, initializing [ 3975.396534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3975.402944] Lustre: 137946:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3975.479681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3975.485370] Lustre: Skipped 1 previous similar message [ 3977.349744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3977.366145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3977.532334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3981.933429] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 3990.110877] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3998.156469] Lustre: lustre-MDD0000: changelog on [ 4000.637742] Lustre: lustre-MDD0001: changelog on [ 5681.222208] Lustre: lustre-MDD0001: changelog off [ 5683.130500] Lustre: 135985:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5684.134797] Lustre: lustre-MDD0000: changelog off [ 5692.898964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5692.900809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5692.908725] LustreError: Skipped 1 previous similar message [ 5692.924268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5692.924275] Lustre: Skipped 2 previous similar messages [ 5695.717879] Lustre: server umount lustre-OST0000 complete [ 5699.559640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5699.568072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.583416] Lustre: Skipped 1 previous similar message [ 5699.587847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5699.594864] Lustre: Skipped 1 previous similar message [ 5703.145129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5708.258851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5708.274962] Lustre: Skipped 3 previous similar messages [ 5713.376251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5713.624666] Lustre: server umount lustre-MDT0000 complete [ 5717.138917] LustreError: 135969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952173 with bad export cookie 13683450415716544460 [ 5717.151229] LustreError: 135969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5717.153159] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5717.546530] Lustre: server umount lustre-MDT0001 complete [ 5727.945561] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 5730.527874] Key type lgssc unregistered [ 5730.780842] LNet: 147062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5730.786547] LNetError: 147062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5730.802339] LNet: Removed LNI 192.168.206.136@tcp [ 5731.667174] Key type .llcrypt unregistered [ 5731.669970] Key type ._llcrypt unregistered [ 5753.594397] Key type ._llcrypt registered [ 5753.598857] Key type .llcrypt registered [ 5753.708681] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 5773.320545] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 5775.066025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5775.115480] alg: No test for adler32 (adler32-zlib) [ 5776.237597] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 5776.548455] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 5778.360673] Key type lgssc registered [ 5780.076988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5819.877326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5819.912823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5821.241176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5821.306600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5821.431883] Lustre: lustre-MDT0000: new disk, initializing [ 5821.517819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5821.535207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5826.071964] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5837.441446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5837.591780] Lustre: 150278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5837.644123] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5837.652089] Lustre: Skipped 1 previous similar message [ 5837.784694] Lustre: lustre-MDT0001: new disk, initializing [ 5837.911043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5837.958743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5837.973403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5842.843828] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5851.480908] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5856.553711] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5862.882393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5863.107946] Lustre: lustre-OST0000: new disk, initializing [ 5863.113491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5863.118989] Lustre: 152246:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5863.193089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5868.349392] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5871.654240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5871.661490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5871.801547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5876.413945] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5881.157844] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_module obdecho/obdecho [ 5882.791168] LustreError: 153247:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5882.801136] LustreError: 153247:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5884.519699] LustreError: 153343:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5897.703389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5897.706557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5897.722543] Lustre: Skipped 2 previous similar messages [ 5897.738723] Lustre: Skipped 2 previous similar messages [ 5900.703625] Lustre: server umount lustre-MDT0000 complete [ 5904.965835] LustreError: 150270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952361 with bad export cookie 1603940796894150261 [ 5904.966442] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5904.994884] LustreError: 150270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5905.287524] Lustre: server umount lustre-MDT0001 complete [ 5921.179844] Lustre: server umount lustre-OST0000 complete [ 5944.663485] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 5953.934454] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 5986.557104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5986.704752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5986.722620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5986.791782] Lustre: lustre-MDT0000: new disk, initializing [ 5986.854751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5986.885487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5990.652994] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 5999.963892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6000.101783] Lustre: 157063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6000.178779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6000.185000] Lustre: Skipped 1 previous similar message [ 6000.259267] Lustre: lustre-MDT0001: new disk, initializing [ 6000.382266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6000.421419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6000.445518] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6004.035488] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6012.046549] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6016.654512] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6022.731680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6023.108808] Lustre: lustre-OST0000: new disk, initializing [ 6023.118360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6023.127590] Lustre: 159032:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6023.233409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6028.814758] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6032.553652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6032.566988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6032.675140] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6039.236879] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6046.383593] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6046.733640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6054.171562] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6054.440451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6058.474434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6058.476211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6058.502318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6058.543987] Lustre: Skipped 1 previous similar message [ 6062.785440] Lustre: server umount lustre-OST0000 complete [ 6068.706446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6068.721134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6068.727118] Lustre: Skipped 2 previous similar messages [ 6068.755336] Lustre: Skipped 1 previous similar message [ 6073.825961] LustreError: 157070:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.849684] LustreError: 157070:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6073.924424] Lustre: server umount lustre-MDT0000 complete [ 6077.846319] LustreError: 157053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952534 with bad export cookie 1603940796894151220 [ 6077.854589] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6077.864638] LustreError: 157053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6078.604101] Lustre: server umount lustre-MDT0001 complete [ 6082.388509] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:48:56 (1790952536) [ 6084.228081] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6086.155597] Lustre: DEBUG MARKER: == conf-sanity test 138: MDT mount failure must not LBUG destroying recovery exports ========================================================== 10:49:00 (1790952540) [ 6103.832484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6104.514819] LustreError: 162002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.611856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6109.061673] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6109.668543] LustreError: 162003:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6114.786225] LustreError: 162002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6116.683399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6121.145889] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6128.176837] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6134.672587] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6142.508473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6142.998323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6143.010396] Lustre: Skipped 1 previous similar message [ 6150.541433] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6158.640923] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6167.027746] Lustre: *** cfs_fail_loc=718, val=0*** [ 6167.244515] Lustre: server umount lustre-MDT0000 complete [ 6168.033892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6168.042464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6168.055177] LustreError: 162003:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6170.499339] Lustre: *** cfs_fail_loc=718, val=0*** [ 6170.501134] Lustre: Skipped 3 previous similar messages [ 6170.502946] LustreError: 161984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952627 with bad export cookie 1603940796894152102 [ 6170.514637] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6170.540637] LustreError: 161984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6170.852164] Lustre: server umount lustre-MDT0001 complete [ 6179.607849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.705124] LustreError: 165326:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 6180.720345] LustreError: 165326:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000d1be65da x1877949528780672/t0(0) o250->MGC192.168.206.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1790952637 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6180.761966] LustreError: 165326:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 6181.057265] Lustre: *** cfs_fail_loc=723, val=0*** [ 6181.059752] Lustre: Skipped 1 previous similar message [ 6181.148369] LustreError: 165355:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-122) [ 6181.155811] LustreError: 165355:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.136@tcp: cfg command failed: rc = -122 [ 6181.160354] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6181.160354] [ 6181.168332] LustreError: MGC192.168.206.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -122. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6181.179739] LustreError: 165326:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -122 [ 6181.188751] LustreError: 165326:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -122 [ 6181.199487] LustreError: 165326:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 6181.206123] LustreError: 165326:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6181.277492] Lustre: server umount lustre-MDT0000 complete [ 6181.280083] LustreError: 165326:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -122 [ 6188.000107] Lustre: 148492:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790952628/real 1790952628] req@00000000eb202292 x1877949528781056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790952644 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6188.032212] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6189.089960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6189.094934] Lustre: 148494:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790952628/real 1790952628] req@0000000060e2e10d x1877949528780928/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790952644 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6189.235555] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6189.525602] LustreError: 163833:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6189.553816] LustreError: 163833:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6191.650100] Lustre: 148491:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790952632/real 1790952632] req@00000000fc124acf x1877949528781184/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790952648 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6193.860935] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6196.704137] Lustre: 148494:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790952637/real 1790952637] req@000000007ff33de9 x1877949528781568/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790952653 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6196.739329] Lustre: 148494:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6198.768570] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6201.596984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6201.825627] Lustre: 148493:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790952642/real 1790952642] req@0000000020660f2c x1877949528783232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790952658 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6201.858858] Lustre: 148493:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6202.116218] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6205.329465] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6206.564246] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6207.471050] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6207.493568] Lustre: lustre-MDT0001: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6213.830224] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6215.538752] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6221.562463] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6222.942807] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6233.076856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6233.081564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6233.081991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6233.106427] Lustre: Skipped 2 previous similar messages [ 6238.184610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6238.192117] Lustre: Skipped 2 previous similar messages [ 6238.971323] Lustre: server umount lustre-OST0000 complete [ 6243.306521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6248.418450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6248.427549] Lustre: Skipped 3 previous similar messages [ 6249.336792] Lustre: server umount lustre-MDT0000 complete [ 6253.539519] LustreError: 165766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6253.582308] LustreError: 165766:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6254.405752] LustreError: 161983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952710 with bad export cookie 1603940796894153019 [ 6254.406968] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6254.418987] LustreError: 161983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6254.950407] Lustre: server umount lustre-MDT0001 complete [ 6268.339950] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 6272.104164] Key type lgssc unregistered [ 6272.465947] LNet: 168582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6272.473224] LNetError: 168582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6272.498195] LNet: Removed LNI 192.168.206.136@tcp [ 6273.564650] Key type .llcrypt unregistered [ 6273.568514] Key type ._llcrypt unregistered [ 6295.225685] Key type ._llcrypt registered [ 6295.227497] Key type .llcrypt registered [ 6295.335303] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 6296.051589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6296.063199] alg: No test for adler32 (adler32-zlib) [ 6297.137789] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 6297.416686] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 6299.120537] Key type lgssc registered [ 6300.145772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6306.939982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6306.958168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6308.432707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6312.582738] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6313.444115] LustreError: 169916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6313.480359] LustreError: 169916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6317.602258] LustreError: 169917:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6320.495946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6320.769069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6324.717432] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6330.836832] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6335.648236] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6341.369891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6341.638271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6347.905991] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6353.903929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6354.416780] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6362.035214] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.36@tcp (stopping) [ 6364.130460] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.132952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6364.151500] Lustre: Skipped 1 previous similar message [ 6365.470452] Lustre: server umount lustre-MDT0000 complete [ 6367.120748] LustreError: 171755:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6368.909459] LustreError: 169896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790952825 with bad export cookie 1969224073222304894 [ 6368.927577] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6369.249739] LustreError: 169913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6369.252221] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6369.269264] Lustre: Skipped 1 previous similar message [ 6369.276324] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6369.283818] Lustre: Skipped 2 previous similar messages [ 6372.288471] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.36@tcp (stopping) [ 6373.282926] LustreError: 169912:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6373.310164] LustreError: 169912:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6375.462474] Lustre: server umount lustre-MDT0001 complete [ 6381.885362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6394.306356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6394.559283] LustreError: 173759:0:(llog.c:1655:llog_backup()) MGC192.168.206.136@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6394.564856] Lustre: 173759:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.136@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6398.946375] LustreError: 173764:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6415.329545] LustreError: 173764:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6415.363755] LustreError: 173764:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 6449.040258] LustreError: 173765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6449.068710] LustreError: 173765:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 22 previous similar messages [ 6455.776351] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6455.781457] Lustre: 173759:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.136@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6496.752448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6501.199666] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6501.865262] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6508.275040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6513.549380] LustreError: 173764:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6513.595462] LustreError: 173764:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 44 previous similar messages [ 6515.188342] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1b5416dc77904e8d [ 6515.195979] Lustre: MGC192.168.206.136@tcp: Connection restored to 0@lo (at 0@lo) [ 6515.566941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6515.668201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6520.833271] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6520.888716] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6533.795690] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6536.901478] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6543.942767] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6545.398507] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6566.881987] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6566.886128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6566.912542] Lustre: Skipped 1 previous similar message [ 6566.920526] Lustre: Skipped 1 previous similar message [ 6569.975859] Lustre: server umount lustre-MDT0000 complete [ 6574.914790] LustreError: 169897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953031 with bad export cookie 1969224073222311565 [ 6574.916597] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6574.928645] LustreError: 169897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6575.013988] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6575.031449] Lustre: Skipped 1 previous similar message [ 6575.043654] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6575.052724] Lustre: Skipped 2 previous similar messages [ 6575.249431] Lustre: server umount lustre-MDT0001 complete [ 6581.581455] Lustre: server umount lustre-OST0000 complete [ 6602.999372] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 6611.947913] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 6647.774247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6648.178349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6648.215629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6648.292874] Lustre: lustre-MDT0000: new disk, initializing [ 6648.387411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6648.414554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6652.547503] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6662.862036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.040931] Lustre: 178869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6663.117637] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6663.122674] Lustre: Skipped 1 previous similar message [ 6663.234231] Lustre: lustre-MDT0001: new disk, initializing [ 6663.369931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6663.389447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6668.306986] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6680.667800] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6689.711079] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6698.846813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6699.289559] Lustre: lustre-OST0000: new disk, initializing [ 6699.296549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6699.303915] Lustre: 180839:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6699.476513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6699.491348] Lustre: Skipped 1 previous similar message [ 6705.477701] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6706.346685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6706.363974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6706.468873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6714.639687] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6719.701832] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6719.879515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6724.097430] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6724.333828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6727.141689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6727.145701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6727.157015] LustreError: Skipped 1 previous similar message [ 6727.158193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6727.193600] Lustre: Skipped 1 previous similar message [ 6732.283971] LustreError: 181231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6732.295886] LustreError: 181231:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 6732.419904] Lustre: server umount lustre-OST0000 complete [ 6737.382659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6737.402861] Lustre: Skipped 1 previous similar message [ 6750.177490] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6750.535886] Lustre: server umount lustre-MDT0000 complete [ 6753.793581] LustreError: 178862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953210 with bad export cookie 1969224073222328624 [ 6753.803381] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6753.808689] LustreError: 178862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6754.140801] Lustre: server umount lustre-MDT0001 complete [ 6762.708572] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:00:16 (1790953216) [ 6769.512433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6770.326589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6776.257262] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6784.843679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6784.993903] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6785.000628] Lustre: Skipped 7 previous similar messages [ 6789.408393] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6796.118783] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6801.545526] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6808.287666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6814.280577] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6820.600248] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6830.986729] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:01:24 (1790953284) [ 6835.688670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6835.694441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6841.107300] Lustre: server umount lustre-OST0000 complete [ 6850.816961] Lustre: server umount lustre-MDT0000 complete [ 6854.226102] LustreError: 183232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953310 with bad export cookie 1969224073222329506 [ 6854.226656] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6854.237822] LustreError: 183232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6854.794637] Lustre: server umount lustre-MDT0001 complete [ 6864.833437] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 6867.142211] Key type lgssc unregistered [ 6867.430505] LNet: 186985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6867.436846] LNetError: 186985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6867.454266] LNet: Removed LNI 192.168.206.136@tcp [ 6868.200695] Key type .llcrypt unregistered [ 6868.202449] Key type ._llcrypt unregistered [ 6883.940951] Key type ._llcrypt registered [ 6883.944700] Key type .llcrypt registered [ 6884.051481] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 6884.982466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6885.163591] alg: No test for adler32 (adler32-zlib) [ 6886.229716] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 6886.446494] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 6888.168182] Key type lgssc registered [ 6888.983130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6896.209971] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6896.229441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6911.712474] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6916.832550] LustreError: 188236:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.136@tcp: failed processing log, type 4: rc = -110 [ 6921.952618] LustreError: 188236:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6921.963808] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6921.967477] LustreError: MGC192.168.206.136@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6921.981823] LustreError: 188236:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6921.990682] LustreError: 188236:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 6921.994834] LustreError: 188236:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6922.047144] Lustre: server umount lustre-OST0000 complete [ 6922.054453] LustreError: 188236:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6932.051420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6933.440617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6937.450900] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6943.714962] LustreError: 188740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6943.737081] LustreError: 188740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6944.443517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6944.747158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6948.179403] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6954.767489] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6959.809454] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6965.333789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6965.499707] LustreError: 190554:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6965.510152] Lustre: 190554:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.136@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6965.653273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6970.988647] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 6978.012786] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6980.585027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6980.596415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6980.618952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6984.169922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6984.171452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6984.182368] Lustre: Skipped 1 previous similar message [ 6984.194371] Lustre: Skipped 2 previous similar messages [ 6986.320852] Lustre: server umount lustre-MDT0000 complete [ 6989.283822] LustreError: 188740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6989.315475] LustreError: 188740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6989.407986] LustreError: 191314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953446 with bad export cookie 6872735842206720945 [ 6989.416850] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6989.423089] LustreError: 191314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6989.778346] Lustre: server umount lustre-MDT0001 complete [ 7004.569853] Lustre: server umount lustre-OST0000 complete [ 7017.068497] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 7029.403316] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7071.947352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7072.246770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7072.278417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7072.358612] Lustre: lustre-MDT0000: new disk, initializing [ 7072.483477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7072.514554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7077.300467] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7087.792946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7087.884987] Lustre: 194079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7087.937313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7087.944195] Lustre: Skipped 1 previous similar message [ 7088.016609] Lustre: lustre-MDT0001: new disk, initializing [ 7088.108853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7088.151538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7088.162164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7092.448186] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7102.218338] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7108.064486] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7115.754825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7116.112930] Lustre: lustre-OST0000: new disk, initializing [ 7116.115719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7116.135548] Lustre: 196052:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7116.274211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7122.542643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7122.582889] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7122.808512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7123.977391] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7136.594582] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7142.179365] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7142.416505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7147.418251] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7147.758278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7153.633989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7153.638556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7153.649886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7153.657970] Lustre: Skipped 1 previous similar message [ 7158.770070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7158.780332] Lustre: Skipped 1 previous similar message [ 7163.877711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7163.902608] Lustre: Skipped 1 previous similar message [ 7164.384147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7164.884540] Lustre: server umount lustre-OST0000 complete [ 7170.024366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7170.038222] LustreError: Skipped 1 previous similar message [ 7170.048450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7170.072737] Lustre: Skipped 1 previous similar message [ 7174.115740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7174.118665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7174.146634] Lustre: Skipped 2 previous similar messages [ 7176.355665] Lustre: server umount lustre-MDT0000 complete [ 7179.241910] LustreError: 194089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7179.261433] LustreError: 194089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7180.408212] LustreError: 194117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953637 with bad export cookie 6872735842206721666 [ 7180.410671] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7180.422883] LustreError: 194117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7180.885534] Lustre: server umount lustre-MDT0001 complete [ 7190.512263] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:07:24 (1790953644) [ 7207.587811] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 7210.197913] Key type lgssc unregistered [ 7210.493824] LNet: 199000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7210.514522] LNetError: 199000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7210.534954] LNet: Removed LNI 192.168.206.136@tcp [ 7211.319405] Key type .llcrypt unregistered [ 7211.329527] Key type ._llcrypt unregistered [ 7227.981031] Key type ._llcrypt registered [ 7227.984744] Key type .llcrypt registered [ 7228.125124] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7228.825359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7228.923481] alg: No test for adler32 (adler32-zlib) [ 7230.048947] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 7230.213561] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 7231.856431] Key type lgssc registered [ 7233.130801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7240.797158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7240.811032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7242.229001] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7246.469858] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7251.426102] LustreError: 200237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7251.439954] LustreError: 200237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7253.649175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7254.020372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7258.105246] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7264.920647] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7269.143102] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7275.758145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7275.968356] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7275.970492] LustreError: 202103:0:(llog.c:1696:llog_backup()) MGC192.168.206.136@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7275.981723] Lustre: 202103:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.136@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7276.067442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7281.169976] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7288.463940] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7292.386725] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7292.392314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7292.397368] Lustre: Skipped 1 previous similar message [ 7292.412642] Lustre: Skipped 2 previous similar messages [ 7296.704703] Lustre: server umount lustre-MDT0000 complete [ 7297.508949] LustreError: 200976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7297.521505] LustreError: 200976:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7300.126289] LustreError: 200218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953756 with bad export cookie 9142291065977166507 [ 7300.131792] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7300.144640] LustreError: 200218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7300.645818] Lustre: server umount lustre-MDT0001 complete [ 7314.976981] Lustre: server umount lustre-OST0000 complete [ 7323.608539] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 7332.320196] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7366.640456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7366.883975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7366.912740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7366.984149] Lustre: lustre-MDT0000: new disk, initializing [ 7367.075708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7367.096426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7371.306309] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7382.851314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.054363] Lustre: 205625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7383.121095] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7383.124453] Lustre: Skipped 1 previous similar message [ 7383.133437] LustreError: 206407:0:(llog.c:1696:llog_backup()) MGC192.168.206.136@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7383.149560] LustreError: 206407:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7383.162664] Lustre: 206407:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.136@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7383.187204] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7383.193216] Lustre: Skipped 1 previous similar message [ 7383.284075] Lustre: lustre-MDT0001: new disk, initializing [ 7383.421697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7383.462569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7383.484274] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7386.999078] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7395.251573] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7399.885940] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7406.072983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7406.253304] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7406.256414] Lustre: Skipped 1 previous similar message [ 7406.259978] LustreError: 207566:0:(llog.c:1696:llog_backup()) MGC192.168.206.136@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7406.267592] LustreError: 207566:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7406.272524] Lustre: 207566:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.136@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7406.284933] Lustre: 207566:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7406.303491] Lustre: lustre-OST0000: new disk, initializing [ 7406.307399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7406.313454] Lustre: 207592:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7406.400884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7411.688078] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7415.381731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7415.394096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7415.581381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7420.768954] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7424.611190] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7424.785362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7428.347598] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7428.596727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7431.137738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7431.151745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7431.166091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7431.172537] Lustre: Skipped 2 previous similar messages [ 7431.190295] Lustre: Skipped 1 previous similar message [ 7436.271280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7436.280282] Lustre: Skipped 1 previous similar message [ 7436.462586] Lustre: server umount lustre-OST0000 complete [ 7439.841234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7439.846450] LustreError: Skipped 1 previous similar message [ 7439.850167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7439.862374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7445.966738] Lustre: server umount lustre-MDT0000 complete [ 7446.497325] LustreError: 205633:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7446.509599] LustreError: 205633:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7449.092410] LustreError: 205617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790953905 with bad export cookie 9142291065977167228 [ 7449.097161] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7449.111438] LustreError: 205617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7449.392407] Lustre: server umount lustre-MDT0001 complete [ 7456.710554] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:11:50 (1790953910) [ 7470.998493] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7480.068038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7480.406381] LustreError: 210718:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7480.463946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7483.460496] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7485.933955] LustreError: 210719:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7490.516191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.206710] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7500.704406] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7508.150721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7514.454718] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7522.669913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7522.895668] Lustre: lustre-OST0001: new disk, initializing [ 7522.900735] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7522.910205] Lustre: 213024:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7522.990140] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7522.997346] Lustre: Skipped 2 previous similar messages [ 7528.764454] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7532.149917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7532.167682] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7532.233177] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7539.068367] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7545.867626] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7547.692635] Lustre: DEBUG MARKER: ADD OST3 [ 7552.474745] loop: module loaded [ 7552.543989] loop0: detected capacity change from 0 to 204800000 [ 7552.666190] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7552.676387] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7552.686440] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7557.994473] loop0: detected capacity change from 0 to 204800000 [ 7559.143651] Lustre: DEBUG MARKER: STOP OST3 [ 7563.139693] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7563.376651] Lustre: lustre-OST0003: new disk, initializing [ 7563.380701] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7563.388245] Lustre: 215011:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7564.901173] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7565.016770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7565.022880] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7565.053061] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7566.947167] LustreError: 215172:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7567.597405] Lustre: Failing over lustre-OST0003 [ 7567.773370] Lustre: server umount lustre-OST0003 complete [ 7569.799376] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7569.848257] LustreError: 212204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7570.422042] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7570.426900] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7570.438641] Lustre: Skipped 2 previous similar messages [ 7578.712577] Lustre: DEBUG MARKER: START OST3 again [ 7586.087185] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7586.554822] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7587.856047] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7588.599507] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7588.609156] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7593.008528] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7612.926880] Lustre: Failing over lustre-OST0003 [ 7613.055535] Lustre: server umount lustre-OST0003 complete [ 7614.439602] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7614.453971] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7614.468513] LustreError: 212204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7614.483900] LustreError: 212204:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 7632.877349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7632.888322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7632.898750] Lustre: Skipped 4 previous similar messages [ 7632.911871] Lustre: Skipped 5 previous similar messages [ 7636.578808] Lustre: server umount lustre-MDT0000 complete [ 7640.157525] LustreError: 213023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954096 with bad export cookie 9142291065977168110 [ 7640.169284] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7640.181030] LustreError: 213023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7643.120756] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7643.132125] Lustre: Skipped 1 previous similar message [ 7646.632266] Lustre: server umount lustre-MDT0001 complete [ 7650.217496] Lustre: server umount lustre-OST0000 complete [ 7653.678845] Lustre: server umount lustre-OST0001 complete [ 7659.879994] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 7667.575334] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7700.120290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7700.367585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7700.408699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7700.473654] Lustre: lustre-MDT0000: new disk, initializing [ 7700.632697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7700.644029] Lustre: Skipped 2 previous similar messages [ 7700.668975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7704.305374] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7714.803315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7714.870987] Lustre: 220488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7715.097347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7720.278229] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7729.456883] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7734.818492] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7740.922629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7741.174330] Lustre: lustre-OST0000: new disk, initializing [ 7741.181375] Lustre: Skipped 1 previous similar message [ 7741.187647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7741.198762] Lustre: Skipped 2 previous similar messages [ 7741.206073] Lustre: 222458:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7743.003074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7743.010537] Lustre: Skipped 1 previous similar message [ 7743.016355] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7743.151911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7747.671560] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7755.503751] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7759.905648] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7760.100503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7763.998245] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7764.187697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7768.547980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7768.551601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7768.558625] LustreError: Skipped 1 previous similar message [ 7768.561689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7768.575884] Lustre: Skipped 3 previous similar messages [ 7768.587537] Lustre: Skipped 3 previous similar messages [ 7780.320358] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7780.495635] Lustre: server umount lustre-OST0000 complete [ 7786.977759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7790.475867] Lustre: server umount lustre-MDT0000 complete [ 7794.007133] LustreError: 220481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954250 with bad export cookie 9142291065977170406 [ 7794.019868] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7794.027766] LustreError: 220481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7794.587547] Lustre: server umount lustre-MDT0001 complete [ 7797.539574] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:17:32 (1790954252) [ 7812.140354] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 7821.783681] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 7859.210600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7859.485817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7859.540319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7859.631944] Lustre: lustre-MDT0000: new disk, initializing [ 7859.757572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7859.765901] Lustre: Skipped 2 previous similar messages [ 7859.789807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7863.555670] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7872.327573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7872.438328] Lustre: 227071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7872.719688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7876.758117] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7885.001571] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7889.995857] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7896.310787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7896.623647] Lustre: 229039:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7898.438147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7902.327746] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7910.839943] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7914.704872] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7914.902647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7919.296000] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7919.567529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7924.209460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7924.212299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7924.218166] LustreError: Skipped 1 previous similar message [ 7924.224204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7924.244882] Lustre: Skipped 3 previous similar messages [ 7924.268063] Lustre: Skipped 8 previous similar messages [ 7927.852938] Lustre: server umount lustre-OST0000 complete [ 7945.184166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7945.323777] Lustre: server umount lustre-MDT0000 complete [ 7948.769911] LustreError: 227082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7948.805176] LustreError: 227082:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 7948.955300] LustreError: 227875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954405 with bad export cookie 9142291065977171281 [ 7948.959091] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7948.964839] LustreError: 227875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7949.266694] Lustre: server umount lustre-MDT0001 complete [ 7956.608781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7960.718325] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7968.650476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7973.949068] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 7981.632432] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7988.402350] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7995.905964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8002.540516] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8009.648128] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8021.751968] Lustre: server umount lustre-OST0000 complete [ 8025.569266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8025.575042] LustreError: Skipped 1 previous similar message [ 8025.583380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8025.591545] Lustre: Skipped 2 previous similar messages [ 8025.602883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8025.617041] Lustre: Skipped 6 previous similar messages [ 8036.407737] Lustre: server umount lustre-MDT0000 complete [ 8039.397144] LustreError: 231254:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8039.444828] LustreError: 231254:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 8039.822218] LustreError: 231236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954496 with bad export cookie 9142291065977172170 [ 8039.830787] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8050.346286] Lustre: server umount lustre-MDT0001 complete [ 8147.838883] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 8150.660445] Key type lgssc unregistered [ 8151.058582] LNet: 234916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8151.067549] LNetError: 234916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8151.098203] LNet: Removed LNI 192.168.206.136@tcp [ 8151.938727] Key type .llcrypt unregistered [ 8151.945820] Key type ._llcrypt unregistered [ 8178.122785] Key type ._llcrypt registered [ 8178.130228] Key type .llcrypt registered [ 8178.260266] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 8199.610507] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 8201.245810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8201.290103] alg: No test for adler32 (adler32-zlib) [ 8202.482845] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 8202.833707] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 8204.536790] Key type lgssc registered [ 8206.044643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8237.672194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8237.709447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8238.907961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8238.934337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8239.039120] Lustre: lustre-MDT0000: new disk, initializing [ 8239.116380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8239.135971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8242.788056] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8252.734944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8252.817255] Lustre: 238309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8252.863086] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8252.869598] Lustre: Skipped 1 previous similar message [ 8252.956655] Lustre: lustre-MDT0001: new disk, initializing [ 8253.051537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8253.081454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8253.091156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8257.767981] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8268.755710] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8275.760713] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8282.517828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8282.761638] Lustre: lustre-OST0000: new disk, initializing [ 8282.766278] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8282.771731] Lustre: 240280:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8282.867746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8289.277404] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8290.889443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8290.899619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8291.044697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8300.545907] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8305.846945] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8306.169789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8311.648873] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8311.860990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8316.900434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8316.915565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8316.932915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8316.935067] Lustre: Skipped 1 previous similar message [ 8320.390576] Lustre: server umount lustre-OST0000 complete [ 8324.581053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8324.597077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8324.643975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8324.658924] Lustre: Skipped 1 previous similar message [ 8327.139155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8327.141822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8327.184028] Lustre: Skipped 1 previous similar message [ 8330.664366] Lustre: server umount lustre-MDT0000 complete [ 8334.785673] LustreError: 238301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954791 with bad export cookie 16988354409470150810 [ 8334.792774] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8334.809104] LustreError: 238301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8335.260268] Lustre: server umount lustre-MDT0001 complete [ 8354.697257] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 8363.749420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8364.057080] LustreError: 243215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8364.079401] LustreError: 243215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8364.121140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8368.113143] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8369.125385] LustreError: 243216:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8374.243507] LustreError: 243215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8375.442096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8375.795606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8379.473380] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8381.767239] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8387.586949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8387.983353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8395.233886] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8404.679269] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8404.826842] Lustre: lustre-OST0001: new disk, initializing [ 8404.834296] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8404.841424] Lustre: 245521:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8404.919270] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8411.900564] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8415.349274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8415.368245] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8415.443340] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8422.777366] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8430.126185] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8444.377119] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:28:18 (1790954898) [ 8447.459715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8447.467421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8447.493077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8451.570563] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8451.575140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8451.594742] Lustre: Skipped 2 previous similar messages [ 8451.604228] Lustre: Skipped 2 previous similar messages [ 8461.280316] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8461.833696] LustreError: 244718:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8461.875659] LustreError: 244718:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8462.041510] Lustre: server umount lustre-MDT0000 complete [ 8466.329279] LustreError: 245520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790954922 with bad export cookie 16988354409470151692 [ 8466.339825] LustreError: 245520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8466.346959] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8466.780864] Lustre: server umount lustre-MDT0001 complete [ 8482.073175] Lustre: server umount lustre-OST0000 complete [ 8483.299183] Lustre: 236523:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790954923/real 1790954923] req@ffff8b834dcf7800 x1877952072630528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790954939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8483.336404] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8486.226838] Lustre: server umount lustre-OST0001 complete [ 8493.818616] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 8507.170611] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 8544.315965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8544.455528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8544.470244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8544.522429] Lustre: lustre-MDT0000: new disk, initializing [ 8544.568857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8544.581522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8548.270593] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8558.729047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8558.867767] Lustre: 249477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8558.955771] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8558.959114] Lustre: Skipped 1 previous similar message [ 8559.020450] Lustre: lustre-MDT0001: new disk, initializing [ 8559.126052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8559.139449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8563.503811] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8571.173763] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8575.030865] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8580.436382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8580.789929] Lustre: lustre-OST0000: new disk, initializing [ 8580.794071] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8580.806242] Lustre: 251447:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8581.972359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8581.981489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8582.163567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8587.386087] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8596.885876] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8601.458322] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8601.630199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8605.178871] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8605.364334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8607.719801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8607.728069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8607.732572] Lustre: Skipped 1 previous similar message [ 8607.738186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8607.738194] Lustre: Skipped 5 previous similar messages [ 8607.776917] LustreError: Skipped 1 previous similar message [ 8613.089231] Lustre: server umount lustre-OST0000 complete [ 8623.573648] Lustre: server umount lustre-MDT0000 complete [ 8626.923061] LustreError: 250275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955083 with bad export cookie 16988354409470153827 [ 8626.925989] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8626.934222] LustreError: 250275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8627.266851] Lustre: server umount lustre-MDT0001 complete [ 8635.452975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8635.860763] LustreError: 253664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8635.918405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8635.925447] Lustre: Skipped 2 previous similar messages [ 8640.542955] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8646.116653] LustreError: 253665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8646.138592] LustreError: 253665:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8649.984260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8654.643509] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8661.939243] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8667.349059] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8673.177416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8678.862984] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 8686.272951] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8833.506755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8833.514206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8833.527511] Lustre: Skipped 2 previous similar messages [ 8833.537370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8833.543403] Lustre: Skipped 7 previous similar messages [ 8845.282161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8845.795806] Lustre: server umount lustre-OST0000 complete [ 8856.816283] Lustre: server umount lustre-MDT0000 complete [ 8860.135846] LustreError: 253660:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8860.165970] LustreError: 253660:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8861.957235] LustreError: 253645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955318 with bad export cookie 16988354409470154709 [ 8861.957904] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8861.969232] LustreError: 253645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8862.325773] Lustre: server umount lustre-MDT0001 complete [ 8952.447566] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 8954.582935] Key type lgssc unregistered [ 8954.831521] LNet: 257274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8954.838408] LNetError: 257274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8954.850109] LNet: Removed LNI 192.168.206.136@tcp [ 8955.642393] Key type .llcrypt unregistered [ 8955.645126] Key type ._llcrypt unregistered [ 8974.986556] Key type ._llcrypt registered [ 8974.988784] Key type .llcrypt registered [ 8975.090353] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 8986.943759] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 8987.837321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8987.886402] alg: No test for adler32 (adler32-zlib) [ 8988.842095] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 8989.002240] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 8990.649141] Key type lgssc registered [ 8991.498907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9026.483087] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 9033.576943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9033.607045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9034.824386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9034.851519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9034.948509] Lustre: lustre-MDT0000: new disk, initializing [ 9035.010307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9035.022293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9038.050307] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9045.876709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9045.972571] Lustre: 261335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9046.026527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9046.030902] Lustre: Skipped 1 previous similar message [ 9046.099753] Lustre: lustre-MDT0001: new disk, initializing [ 9046.162052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9046.191556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9046.205622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9049.124370] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9052.434794] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9056.864805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9057.041222] Lustre: lustre-OST0000: new disk, initializing [ 9057.045682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9057.050906] Lustre: 262967:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9057.121138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9057.643892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9057.652716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9057.728702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9061.539840] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9069.192356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9069.320221] Lustre: lustre-OST0001: new disk, initializing [ 9069.322845] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9069.330307] Lustre: 263840:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9069.399786] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9073.559342] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9075.274088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9075.284717] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9075.338808] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9082.924729] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9088.476477] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9096.167273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9096.171975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9096.174976] Lustre: Skipped 3 previous similar messages [ 9096.180991] Lustre: Skipped 3 previous similar messages [ 9097.216330] Lustre: server umount lustre-MDT0000 complete [ 9099.837826] LustreError: 261327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955556 with bad export cookie 18424257809361027866 [ 9099.840107] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9099.856817] LustreError: 261327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9100.128354] Lustre: server umount lustre-MDT0001 complete [ 9113.298344] Lustre: server umount lustre-OST0000 complete [ 9125.652533] Lustre: server umount lustre-OST0001 complete [ 9137.335651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9145.388699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9151.661499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9158.706823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9171.294638] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 9178.275095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.290235] Lustre: lustre-MDT0000: reset Object Index mappings [ 9178.645796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9181.413367] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9183.718987] LustreError: 268843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9183.733455] LustreError: 268843:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9186.727124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9186.738186] Lustre: lustre-MDT0001: reset Object Index mappings [ 9186.967693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9189.668877] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9191.639573] Lustre: 269986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9195.935737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9197.223913] LustreError: 270337:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9199.611149] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9202.661465] LustreError: 270461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9203.703487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9205.105394] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9205.290380] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9205.294875] Lustre: Skipped 1 previous similar message [ 9209.150778] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9210.357653] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9214.244395] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9216.576470] Lustre: 271855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9220.582285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9220.584961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9220.596907] Lustre: Skipped 3 previous similar messages [ 9220.601880] Lustre: Skipped 2 previous similar messages [ 9225.520591] Lustre: server umount lustre-MDT0000 complete [ 9225.701282] LustreError: 270951:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9225.713327] LustreError: 270951:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9228.307978] LustreError: 268823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955684 with bad export cookie 18424257809361034656 [ 9228.312406] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9228.329680] LustreError: 268823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9228.549930] Lustre: server umount lustre-MDT0001 complete [ 9241.809297] Lustre: server umount lustre-OST0000 complete [ 9254.999551] Lustre: server umount lustre-OST0001 complete [ 9269.189623] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:42:03 (1790955723) [ 9282.316729] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 9289.672738] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 9321.734845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9321.919713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9321.946568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9322.011638] Lustre: lustre-MDT0000: new disk, initializing [ 9322.070866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9322.082251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9325.124862] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9332.749509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9332.826989] Lustre: 276109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9332.871520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9332.874072] Lustre: Skipped 1 previous similar message [ 9332.956246] Lustre: lustre-MDT0001: new disk, initializing [ 9333.079552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9333.091777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9335.948883] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9342.710890] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9346.691083] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9351.422941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9351.618573] Lustre: lustre-OST0000: new disk, initializing [ 9351.621663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9351.626228] Lustre: 278073:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9353.216653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9353.224892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9353.334074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9355.829875] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9362.720418] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9365.933706] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9366.073121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9368.655377] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9368.802269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9373.672070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9373.672284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9373.683899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9373.687611] Lustre: Skipped 2 previous similar messages [ 9376.117232] Lustre: server umount lustre-OST0000 complete [ 9378.786584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9378.792380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9378.796420] Lustre: Skipped 2 previous similar messages [ 9383.905549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9383.912818] Lustre: Skipped 2 previous similar messages [ 9384.743209] Lustre: server umount lustre-MDT0000 complete [ 9387.101154] LustreError: 276099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955843 with bad export cookie 18424257809361036742 [ 9387.106506] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9387.109979] LustreError: 276099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9387.372504] Lustre: server umount lustre-MDT0001 complete [ 9399.661871] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 9405.737229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9405.993385] LustreError: 281002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9406.020378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9406.023522] Lustre: Skipped 2 previous similar messages [ 9408.441700] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9413.431477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9415.934490] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9417.486674] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9421.259371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9425.671730] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9430.779735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9430.891705] Lustre: lustre-OST0001: new disk, initializing [ 9430.894638] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9430.902153] Lustre: 283307:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9432.570412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9432.577819] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9432.650083] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9434.756836] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9441.136979] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9443.209291] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9446.196537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9446.216347] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9453.032179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9453.034340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9453.052985] Lustre: Skipped 3 previous similar messages [ 9453.062614] Lustre: Skipped 2 previous similar messages [ 9457.589814] Lustre: server umount lustre-MDT0000 complete [ 9458.153885] LustreError: 281945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9458.171821] LustreError: 281945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9460.443981] LustreError: 280982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790955917 with bad export cookie 18424257809361037624 [ 9460.446967] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9460.455611] LustreError: 280982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9460.721599] Lustre: server umount lustre-MDT0001 complete [ 9472.739479] Lustre: server umount lustre-OST0000 complete [ 9486.378355] Lustre: server umount lustre-OST0001 complete [ 9498.553289] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 9504.397921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9504.623211] LustreError: 286645:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9507.012579] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9512.116587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9514.885403] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9516.789594] Lustre: 287787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9520.690474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9524.801873] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9530.112899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9531.379779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9531.409181] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9534.307332] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9539.212329] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9541.379679] Lustre: 289662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9544.481213] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9544.537776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9554.857976] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:46:49 (1790956009) [ 9568.738676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9568.746819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9568.759642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9568.766969] Lustre: Skipped 1 previous similar message [ 9574.225701] Lustre: server umount lustre-MDT0000 complete [ 9576.823858] LustreError: 286627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956033 with bad export cookie 18424257809361044001 [ 9576.828947] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9576.834854] LustreError: 286627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9577.440949] LustreError: 286640:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9577.454135] LustreError: 286640:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9591.264220] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9591.430530] Lustre: server umount lustre-MDT0001 complete [ 9608.160136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9608.242985] Lustre: server umount lustre-OST0000 complete [ 9625.056124] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9625.182818] Lustre: server umount lustre-OST0001 complete [ 9629.679210] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [ 9636.361427] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 9664.296828] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 9669.887955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9670.022781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9670.038808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9670.087786] Lustre: lustre-MDT0000: new disk, initializing [ 9670.135141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9670.141352] Lustre: Skipped 7 previous similar messages [ 9670.149803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9672.360402] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9679.681222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9679.762267] Lustre: 294188:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9680.026549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9682.945417] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9690.639086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9690.805807] Lustre: 295819:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9692.970954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9694.782607] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9701.454123] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9701.578977] Lustre: 296688:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9703.101516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9705.605646] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9712.107642] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9714.431856] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9714.435321] Lustre: Skipped 1 previous similar message [ 9723.013819] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:49:37 (1790956177) [ 9725.921777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9725.926761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9725.934871] Lustre: Skipped 5 previous similar messages [ 9725.938541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9725.942247] Lustre: Skipped 10 previous similar messages [ 9730.325484] Lustre: server umount lustre-MDT0000 complete [ 9732.284499] LustreError: 294982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956188 with bad export cookie 18424257809361051946 [ 9732.285988] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9732.293577] LustreError: 294982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9732.509071] Lustre: server umount lustre-MDT0001 complete [ 9744.954795] Lustre: server umount lustre-OST0000 complete [ 9756.815476] Lustre: server umount lustre-OST0001 complete [ 9763.300090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9763.769637] LustreError: 298933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9763.789801] LustreError: 298933:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9767.114053] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9772.965075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9775.415348] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9780.501993] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9784.743993] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9790.246805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9794.954299] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9800.087598] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9821.289905] loop1: detected capacity change from 0 to 204800000 [ 9821.405483] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9821.413089] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9821.429379] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9825.026962] loop1: detected capacity change from 0 to 204800000 [ 9829.344583] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9829.447922] LustreError: 298922:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs14667: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9829.456309] LustreError: 298922:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs14667 [ 9829.459908] LustreError: 298922:0:(mgs_handler.c:517:mgs_target_reg()) fs14667-MDT0001: Error checking target index: rc = -13 [ 9829.468253] LustreError: fs14667-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9829.473849] LustreError: 302673:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 9829.478592] LustreError: 302673:0:(tgt_mount.c:2128:server_put_super()) no obd fs14667-MDT0001 [ 9829.482588] LustreError: 302673:0:(tgt_mount.c:116:server_deregister_mount()) fs14667-MDT0001 not registered [ 9829.503971] Lustre: server umount fs14667-MDT0001 complete [ 9829.506291] LustreError: 302673:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9835.217743] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9835.274522] Lustre: Setting parameter fs14667-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs14667-MDT0001 [ 9835.308627] Lustre: srv-fs14667-MDT0001: No data found on store. Initialize space. [ 9835.313320] Lustre: Skipped 4 previous similar messages [ 9835.371898] Lustre: fs14667-MDT0001: new disk, initializing [ 9835.374202] Lustre: Skipped 3 previous similar messages [ 9835.442638] Lustre: srv-fs14667-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9835.449077] Lustre: 303139:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14667-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9835.453294] LustreError: 303139:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14667-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9838.026838] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9843.207904] Lustre: server umount fs14667-MDT0001 complete [ 9851.678242] loop1: detected capacity change from 0 to 204800000 [ 9855.101750] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9855.306836] Lustre: 304708:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14667-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9855.317151] LustreError: 304708:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14667-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9858.172253] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9861.590711] Lustre: server umount fs14667-MDT0001 complete [ 9870.305979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9870.309866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9870.311694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9870.322976] Lustre: Skipped 5 previous similar messages [ 9870.327305] Lustre: Skipped 5 previous similar messages [ 9874.092859] Lustre: server umount lustre-MDT0000 complete [ 9879.325409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9879.403124] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9879.511547] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9882.334720] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9884.653475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9887.167805] loop1: detected capacity change from 0 to 204800000 [ 9890.296532] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9890.511632] Lustre: 307155:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs14667-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9890.519924] LustreError: 307155:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs14667-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9892.749949] LustreError: 300760:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9892.760603] LustreError: 300760:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 56 previous similar messages [ 9893.006425] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9895.635307] Lustre: server umount fs14667-MDT0001 complete [ 9910.246287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9910.252033] LustreError: Skipped 1 previous similar message [ 9913.082474] Lustre: server umount lustre-OST0000 complete [ 9921.660565] Lustre: server umount lustre-MDT0000 complete [ 9923.976285] LustreError: 298915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956380 with bad export cookie 18424257809361054627 [ 9923.980154] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9923.984061] LustreError: 298915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9924.167812] Lustre: server umount lustre-MDT0001 complete [ 9931.205437] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [ 9933.066230] Key type lgssc unregistered [ 9933.271960] LNet: 309494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9933.276683] LNetError: 309494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9933.288372] LNet: Removed LNI 192.168.206.136@tcp [ 9933.843152] Key type .llcrypt unregistered [ 9933.847474] Key type ._llcrypt unregistered [ 9947.255839] Key type ._llcrypt registered [ 9947.258647] Key type .llcrypt registered [ 9947.333059] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [ 9947.840351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9947.868107] alg: No test for adler32 (adler32-zlib) [ 9948.830608] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [ 9948.970170] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [ 9950.584328] Key type lgssc registered [ 9951.329831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9957.098784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9957.116172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9958.545895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9961.432535] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9966.902771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9967.183034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9969.782683] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9974.202598] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9977.327757] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9982.020394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9982.291067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9984.362826] LustreError: 312587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9984.381124] LustreError: 312587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9986.011213] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [ 9989.608231] LustreError: 312588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9990.382740] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9991.652673] LustreError: 312775:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9991.670166] LustreError: 312775:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9996.773797] LustreError: 312588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9996.787327] LustreError: 312588:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10001.893386] LustreError: 312588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10001.905653] LustreError: 312588:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10002.015848] loop1: detected capacity change from 0 to 204800000 [10002.148691] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10002.157735] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10002.165744] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10005.222578] loop1: detected capacity change from 0 to 204800000 [10008.565319] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10008.650540] Lustre: fs233-OST0000: new disk, initializing [10008.653679] Lustre: srv-fs233-OST0000: No data found on store. Initialize space. [10008.657917] Lustre: 314490:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10008.697127] Lustre: fs233-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10012.131973] LustreError: 312587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10012.143104] LustreError: 312587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10012.259859] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10017.440746] Lustre: server umount fs233-OST0000 complete [10024.563737] loop1: detected capacity change from 0 to 204800000 [10028.096366] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10028.243438] Lustre: fs233-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10028.943773] LustreError: 312588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10028.958107] LustreError: 312588:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [10031.792582] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10036.783432] loop2: detected capacity change from 0 to 204800000 [10036.901733] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10036.909860] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10036.916444] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10040.201130] loop2: detected capacity change from 0 to 204800000 [10043.910662] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10043.960460] LustreError: fs233-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10043.971772] LustreError: fs233-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10043.982324] LustreError: 317526:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10043.989575] LustreError: 317526:0:(tgt_mount.c:2128:server_put_super()) no obd fs233-OST0001 [10043.994188] LustreError: 317526:0:(tgt_mount.c:116:server_deregister_mount()) fs233-OST0001 not registered [10044.013659] Lustre: server umount fs233-OST0001 complete [10044.016707] LustreError: 317526:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10051.204516] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10051.414499] Lustre: fs233-OST0001: new disk, initializing [10051.424766] Lustre: srv-fs233-OST0001: No data found on store. Initialize space. [10051.436447] Lustre: 318079:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10051.510143] Lustre: fs233-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10056.713204] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10063.332291] LustreError: 312775:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10063.359819] LustreError: 312775:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [10063.636962] Lustre: server umount fs233-OST0000 complete [10068.453601] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10070.032818] Lustre: server umount fs233-OST0001 complete [10077.407976] loop1: detected capacity change from 0 to 204800000 [10078.711564] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10081.740480] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10081.903389] Lustre: fs233-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10085.668607] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10090.139725] loop2: detected capacity change from 0 to 204800000 [10093.924168] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10098.657876] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10105.128192] Lustre: server umount fs233-OST0000 complete [10111.411212] Lustre: server umount fs233-OST0001 complete [10122.722649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10122.734285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10122.743964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10124.771461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10124.773237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10124.776767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10126.306929] Lustre: server umount lustre-OST0000 complete [10129.897109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10129.903740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10129.914144] Lustre: Skipped 1 previous similar message [10129.928478] Lustre: Skipped 2 previous similar messages [10134.895981] Lustre: server umount lustre-MDT0000 complete [10135.014688] LustreError: 310761:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10135.031289] LustreError: 310761:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 36 previous similar messages [10137.627664] LustreError: 318078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956594 with bad export cookie 1324315053814493872 [10137.636248] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10137.640931] LustreError: 318078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10137.865775] Lustre: server umount lustre-MDT0001 complete [10145.223311] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [10147.143836] Key type lgssc unregistered [10147.342747] LNet: 323995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10147.350445] LNetError: 323995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10147.364057] LNet: Removed LNI 192.168.206.136@tcp [10147.875230] Key type .llcrypt unregistered [10147.878291] Key type ._llcrypt unregistered [10168.312246] Key type ._llcrypt registered [10168.313835] Key type .llcrypt registered [10168.370776] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [10177.804401] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [10178.249423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10178.270229] alg: No test for adler32 (adler32-zlib) [10179.175709] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [10179.319058] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [10180.928558] Key type lgssc registered [10181.597394] Lustre: Echo OBD driver; http://www.lustre.org/ [10205.326676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10205.361180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10206.486137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10206.508857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10206.603771] Lustre: lustre-MDT0000: new disk, initializing [10206.672631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10206.687877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10209.398024] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10216.846397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10216.943985] Lustre: 327846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10217.001884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10217.009271] Lustre: Skipped 1 previous similar message [10217.084704] Lustre: lustre-MDT0001: new disk, initializing [10217.180300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10217.219235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10217.229129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10220.200401] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10226.974750] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10230.795737] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10235.173968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10235.373846] Lustre: lustre-OST0000: new disk, initializing [10235.376868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10235.381617] Lustre: 329811:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10235.454874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10239.602915] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10241.055611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10241.066858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10241.116407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10245.972180] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10247.184237] Lustre: Mounted lustre-client - version 2.17.59_3_g5d853fc [10247.802518] Lustre: Unmounted lustre-client [10256.355762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10256.358036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10256.363517] LustreError: Skipped 1 previous similar message [10256.367607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10256.372447] Lustre: Skipped 1 previous similar message [10256.383386] Lustre: Skipped 1 previous similar message [10257.384088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10257.387724] Lustre: Skipped 1 previous similar message [10259.207452] Lustre: server umount lustre-OST0000 complete [10262.498076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10262.502049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10262.509305] Lustre: Skipped 1 previous similar message [10262.517453] Lustre: Skipped 1 previous similar message [10267.542899] Lustre: server umount lustre-MDT0000 complete [10269.771225] LustreError: 327839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956726 with bad export cookie 5762921586323716807 [10269.774350] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10269.778532] LustreError: 327839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10269.995714] Lustre: server umount lustre-MDT0001 complete [10275.861835] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [10277.413890] Key type lgssc unregistered [10297.120163] LNet: 1 local NIs in recovery (showing 1): 192.168.206.136@tcp [10298.080377] LNet: 331970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10298.083751] LNetError: 331970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10376.934418] LNet: Removed LNI 192.168.206.136@tcp [10377.333673] Key type .llcrypt unregistered [10377.335195] Key type ._llcrypt unregistered [10385.090921] Key type ._llcrypt registered [10385.092466] Key type .llcrypt registered [10385.139800] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [10392.701275] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [10393.064404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10393.075491] alg: No test for adler32 (adler32-zlib) [10393.949580] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [10394.053599] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [10395.640209] Key type lgssc registered [10396.151460] Lustre: Echo OBD driver; http://www.lustre.org/ [10408.633942] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:01:03 (1790956863) [10411.429878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10411.457875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10412.573791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10412.587680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10412.628919] Lustre: lustre-MDT0000: new disk, initializing [10412.663942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10412.672276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10414.331740] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10419.495222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10419.566853] Lustre: 335192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10419.597557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10419.600444] Lustre: Skipped 1 previous similar message [10419.639535] Lustre: lustre-MDT0001: new disk, initializing [10419.681301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10419.698895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10419.706199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10421.446168] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10426.308802] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10428.668036] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10431.686561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10431.803593] Lustre: lustre-OST0000: new disk, initializing [10431.806299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10431.810773] Lustre: 337162:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10431.847029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10434.313930] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10439.090383] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10440.199856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10440.204816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10440.243708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10453.150340] Lustre: server umount lustre-OST0000 complete [10455.523547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10455.524153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10455.525043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10455.532119] Lustre: Skipped 1 previous similar message [10460.641665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10460.644848] Lustre: Skipped 2 previous similar messages [10464.767150] Lustre: server umount lustre-MDT0000 complete [10466.284819] LustreError: 335990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956922 with bad export cookie 5297611436835407998 [10466.285739] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10466.290259] LustreError: 335990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10475.560338] Lustre: server umount lustre-MDT0001 complete [10480.199807] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [10481.419822] Key type lgssc unregistered [10481.566711] LNet: 339068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10481.570704] LNetError: 339068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10481.583498] LNet: Removed LNI 192.168.206.136@tcp [10481.961338] Key type .llcrypt unregistered [10481.962836] Key type ._llcrypt unregistered [10491.399198] Key type ._llcrypt registered [10491.400437] Key type .llcrypt registered [10491.439785] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [10491.815764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10491.855217] alg: No test for adler32 (adler32-zlib) [10492.737738] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [10492.851305] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [10494.440224] Key type lgssc registered [10494.929091] Lustre: Echo OBD driver; http://www.lustre.org/ [10498.208043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10498.219753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10499.416278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10501.169440] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10504.714670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10504.864565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10506.530700] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10509.726402] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10511.953783] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10514.816634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10514.980529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10517.367872] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10520.465390] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10530.693583] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10531.321324] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10534.374583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10534.380900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10534.384246] Lustre: Skipped 1 previous similar message [10537.364416] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.36@tcp (stopping) [10537.367895] Lustre: Skipped 2 previous similar messages [10539.235121] Lustre: server umount lustre-MDT0000 complete [10539.490154] LustreError: 340335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10539.496524] LustreError: 340335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10540.713205] LustreError: 342176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790956997 with bad export cookie 13497677356928210531 [10540.716495] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10540.719397] LustreError: 342176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10542.474578] LustreError: 342823:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10542.485926] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.36@tcp (stopping) [10544.610767] LustreError: 340330:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10544.611598] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10544.631288] Lustre: Skipped 1 previous similar message [10544.634071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10547.597381] LustreError: 340330:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10547.614630] LustreError: 340330:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10550.754842] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10550.760230] Lustre: Skipped 2 previous similar messages [10552.717856] LustreError: 340331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.729940] LustreError: 340331:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10554.848124] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10554.969652] Lustre: server umount lustre-MDT0001 complete [10562.662952] Lustre: server umount lustre-OST0000 complete [10566.506130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10568.383563] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10569.635706] Lustre: 344511:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.136@tcp [10570.683986] Lustre: server umount MGS complete [10574.708077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10574.935392] LustreError: 345082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10574.981177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10576.778365] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10580.768269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10580.936432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10582.644193] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10586.441354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10589.270890] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10592.360823] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10592.984238] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10595.214161] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10606.299175] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec [10610.146102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10610.147229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10610.151394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10610.158034] Lustre: Skipped 1 previous similar message [10610.161027] Lustre: Skipped 1 previous similar message [10613.651943] Lustre: server umount lustre-OST0000 complete [10615.266745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10615.273388] Lustre: Skipped 1 previous similar message [10621.170964] Lustre: server umount lustre-MDT0000 complete [10622.666438] LustreError: 345063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957079 with bad export cookie 13497677356928211357 [10622.670288] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10622.671216] LustreError: 345063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10622.796315] Lustre: server umount lustre-MDT0001 complete [10627.252105] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [10628.514462] Key type lgssc unregistered [10628.669897] LNet: 348537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10628.673695] LNetError: 348537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10628.687391] LNet: Removed LNI 192.168.206.136@tcp [10629.050425] Key type .llcrypt unregistered [10629.051605] Key type ._llcrypt unregistered [10636.333281] Key type ._llcrypt registered [10636.334870] Key type .llcrypt registered [10636.377461] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [10643.803908] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [10644.188984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10644.245663] alg: No test for adler32 (adler32-zlib) [10645.108186] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [10645.203669] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [10646.792271] Key type lgssc registered [10647.237618] Lustre: Echo OBD driver; http://www.lustre.org/ [10661.077714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10661.092905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10662.172860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10662.185520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10662.222573] Lustre: lustre-MDT0000: new disk, initializing [10662.246795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10662.253282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10663.739894] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10668.087200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10668.130769] Lustre: 351685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10668.156989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10668.158733] Lustre: Skipped 1 previous similar message [10668.196535] Lustre: lustre-MDT0001: new disk, initializing [10668.230782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10668.244599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10668.248525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10669.707159] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10673.656780] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10675.582178] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10677.934051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10678.041845] Lustre: lustre-OST0000: new disk, initializing [10678.044510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10678.048384] Lustre: 353652:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10678.085742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10680.286396] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10684.441513] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10686.034599] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10686.456369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10686.460210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10686.491196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10687.140663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [10688.614614] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10688.694524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10691.555982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10691.556231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10691.557119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10691.561388] Lustre: Skipped 1 previous similar message [10695.514372] Lustre: server umount lustre-OST0000 complete [10699.234322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10699.238202] LustreError: Skipped 1 previous similar message [10699.240566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10699.247954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10699.250892] Lustre: Skipped 1 previous similar message [10701.793745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10701.794154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10703.151116] Lustre: server umount lustre-MDT0000 complete [10704.788342] LustreError: 351679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957161 with bad export cookie 15887881906973520164 [10704.791860] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10704.793600] LustreError: 351679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10704.946485] Lustre: server umount lustre-MDT0001 complete [10708.652412] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:06:03 (1790957163) [10714.866089] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [10719.222853] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [10733.574210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10733.675077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10733.690505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10733.735053] Lustre: lustre-MDT0000: new disk, initializing [10733.766237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10733.776088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10735.385977] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10739.889114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10739.937142] Lustre: 358413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10739.965084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10739.967663] Lustre: Skipped 1 previous similar message [10740.007936] Lustre: lustre-MDT0001: new disk, initializing [10740.049529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10740.065519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10740.071141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10741.613602] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10745.739518] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10747.799628] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10750.477550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10750.608957] Lustre: lustre-OST0000: new disk, initializing [10750.612175] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10750.617831] Lustre: 360382:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10750.656988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10752.952800] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10757.174959] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10758.139503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10758.143943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10758.176467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10773.476148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10773.476512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10773.479595] LustreError: Skipped 1 previous similar message [10773.486184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10773.489885] Lustre: Skipped 2 previous similar messages [10778.594566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10778.597643] Lustre: Skipped 1 previous similar message [10779.299112] Lustre: server umount lustre-OST0000 complete [10781.152907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10781.156760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10781.163164] Lustre: Skipped 1 previous similar message [10786.971830] Lustre: server umount lustre-MDT0000 complete [10788.365114] LustreError: 358405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957244 with bad export cookie 15887881906973521046 [10788.368513] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10788.369246] LustreError: 358405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10788.495982] Lustre: server umount lustre-MDT0001 complete [10792.520492] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [10793.669622] Key type lgssc unregistered [10793.816492] LNet: 362527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10793.818973] LNetError: 362527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10793.830363] LNet: Removed LNI 192.168.206.136@tcp [10794.179541] Key type .llcrypt unregistered [10794.181224] Key type ._llcrypt unregistered [10802.745570] Key type ._llcrypt registered [10802.746978] Key type .llcrypt registered [10802.796495] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [10803.159444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10803.189744] alg: No test for adler32 (adler32-zlib) [10804.059334] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [10804.159922] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [10805.752181] Key type lgssc registered [10806.187501] Lustre: Echo OBD driver; http://www.lustre.org/ [10809.091269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10809.099592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10810.250195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10811.954925] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10815.309237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10815.464729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10817.071166] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10819.813141] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10822.023555] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10824.477083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10824.602174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10826.855667] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [10829.555886] Lustre: DEBUG MARKER: oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10860.514796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10860.516521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10860.516744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10860.519435] Lustre: Skipped 1 previous similar message [10862.996956] Lustre: server umount lustre-OST0000 complete [10865.634314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10865.635197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10865.638353] Lustre: Skipped 1 previous similar message [10865.641339] Lustre: Skipped 1 previous similar message [10870.548092] Lustre: server umount lustre-MDT0000 complete [10870.753932] LustreError: 363793:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10870.761937] LustreError: 363793:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10871.863224] LustreError: 363776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957328 with bad export cookie 7157256093300884808 [10871.865530] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10871.868472] LustreError: 363776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10872.001270] Lustre: server umount lustre-MDT0001 complete [10876.036974] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [10877.138453] Key type lgssc unregistered [10877.275613] LNet: 369415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10877.279415] LNetError: 369415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10877.288486] LNet: Removed LNI 192.168.206.136@tcp [10877.649483] Key type .llcrypt unregistered [10877.651043] Key type ._llcrypt unregistered [10913.362985] Key type ._llcrypt registered [10913.364476] Key type .llcrypt registered [10913.412402] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [10913.546603] Key type .llcrypt unregistered [10913.548638] Key type ._llcrypt unregistered [10943.820821] Key type ._llcrypt registered [10943.822202] Key type .llcrypt registered [10943.866235] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [10944.013452] Key type .llcrypt unregistered [10944.015634] Key type ._llcrypt unregistered [10955.341367] Key type ._llcrypt registered [10955.342864] Key type .llcrypt registered [10955.380024] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [10955.483333] Key type .llcrypt unregistered [10955.484720] Key type ._llcrypt unregistered [10967.417672] Key type ._llcrypt registered [10967.419334] Key type .llcrypt registered [10967.469310] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [10967.604531] Key type .llcrypt unregistered [10967.606107] Key type ._llcrypt unregistered [10997.513931] Key type ._llcrypt registered [10997.515396] Key type .llcrypt registered [10997.559701] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [10997.673488] Key type .llcrypt unregistered [10997.675296] Key type ._llcrypt unregistered [11009.730432] Key type ._llcrypt registered [11009.731985] Key type .llcrypt registered [11009.774390] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [11009.909259] Key type .llcrypt unregistered [11009.910961] Key type ._llcrypt unregistered [11021.255354] Key type ._llcrypt registered [11021.256522] Key type .llcrypt registered [11021.301985] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [11021.416624] Key type .llcrypt unregistered [11021.418346] Key type ._llcrypt unregistered [11033.686403] Key type ._llcrypt registered [11033.687879] Key type .llcrypt registered [11033.736935] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [11033.865355] Key type .llcrypt unregistered [11033.866794] Key type ._llcrypt unregistered [11045.602685] Key type ._llcrypt registered [11045.604378] Key type .llcrypt registered [11045.643284] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [11045.748273] Key type .llcrypt unregistered [11045.750140] Key type ._llcrypt unregistered [11056.983982] Key type ._llcrypt registered [11056.985041] Key type .llcrypt registered [11057.023240] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [11057.142221] Key type .llcrypt unregistered [11057.143657] Key type ._llcrypt unregistered [11078.926541] Key type ._llcrypt registered [11078.928087] Key type .llcrypt registered [11078.971609] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [11079.094255] Key type .llcrypt unregistered [11079.095566] Key type ._llcrypt unregistered [11090.512276] Key type ._llcrypt registered [11090.513824] Key type .llcrypt registered [11090.560646] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [11090.677425] Key type .llcrypt unregistered [11090.679098] Key type ._llcrypt unregistered [11102.089706] Key type ._llcrypt registered [11102.091423] Key type .llcrypt registered [11102.128854] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [11102.229341] Key type .llcrypt unregistered [11102.230447] Key type ._llcrypt unregistered [11121.248099] Key type ._llcrypt registered [11121.249674] Key type .llcrypt registered [11121.288257] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [11127.488409] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [11127.811035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11127.849873] alg: No test for adler32 (adler32-zlib) [11128.695911] Lustre: Lustre: Build Version: 2.17.59_3_g5d853fc [11128.780425] LNet: Added LNI 192.168.206.136@tcp [8/256/0/180] [11130.360175] Key type lgssc registered [11130.746965] Lustre: Echo OBD driver; http://www.lustre.org/ [11142.137283] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11142.146210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11143.223722] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [11143.234551] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [11143.270266] Lustre: tst250fs-MDT0000: new disk, initializing [11143.291415] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11143.297451] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11144.620469] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11148.760849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11148.802715] Lustre: 389308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11148.829529] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [11148.832271] Lustre: Skipped 1 previous similar message [11148.867807] Lustre: tst250fs-MDT0001: new disk, initializing [11148.904615] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11148.918179] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11148.922732] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11150.295112] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11152.612355] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11154.769919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11154.884393] Lustre: tst250fs-OST0000: new disk, initializing [11154.886512] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [11154.889682] Lustre: 390938:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11154.920286] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11156.794813] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11156.796075] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11156.801514] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11156.839984] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11160.930343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11160.994570] Lustre: tst250fs-OST0001: new disk, initializing [11160.996583] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [11160.999681] Lustre: 391809:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11161.034288] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11162.989919] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_default_debug -1 all [11165.461940] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [11166.105696] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11166.110789] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11166.167931] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11176.420223] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11176.422304] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11176.426048] Lustre: Skipped 2 previous similar messages [11176.428462] Lustre: Skipped 1 previous similar message [11181.537439] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11181.540463] Lustre: Skipped 4 previous similar messages [11186.657965] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11186.660544] Lustre: Skipped 3 previous similar messages [11190.240133] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11190.303424] Lustre: server umount tst250fs-MDT0000 complete [11191.638340] LustreError: 389300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790957648 with bad export cookie 13671772572608432939 [11191.640367] LustreError: MGC192.168.206.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11191.643574] LustreError: 389300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11191.775667] Lustre: server umount tst250fs-MDT0001 complete [11203.391612] Lustre: server umount tst250fs-OST0000 complete [11214.952596] Lustre: server umount tst250fs-OST0001 complete [11221.436343] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [11224.374069] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [11236.115834] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:14:51 (1790957691) [11236.724668] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [11242.661473] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing set_hostid [11246.457533] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing load_modules_local [11257.833852] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10977 sec ========= 12:15:13 (1790957713) [11258.453506] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:15:13 (1790957713) === [11259.691961] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:15:14 (1790957714) === [11274.075149] Lustre: DEBUG MARKER: oleg636-server.virtnet: executing unload_modules_local [11275.208247] Key type lgssc unregistered [11275.349257] LNet: 400939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11275.351896] LNetError: 400939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11275.360321] LNet: Removed LNI 192.168.206.136@tcp [11275.694212] Key type .llcrypt unregistered [11275.695526] Key type ._llcrypt unregistered