[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 498344850 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002373] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006736] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010140] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012035] SELinux: Initializing. [ 0.013061] *** VALIDATE selinux *** [ 0.021063] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026235] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029107] *** VALIDATE tmpfs *** [ 0.030420] *** VALIDATE proc *** [ 0.032152] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034251] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041287] debug: unmapping init [mem 0xffffffffbde59000-0xffffffffbde60fff] [ 0.043180] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044645] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045085] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048012] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051011] ... event mask: 000000070000000f [ 0.053211] rcu: Hierarchical SRCU implementation. [ 0.055312] smp: Bringing up secondary CPUs ... [ 0.056570] x86: Booting SMP configuration: [ 0.057024] .... node #0, CPUs: #1 #2 #3 [ 0.061144] smp: Brought up 1 node, 4 CPUs [ 0.063022] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.267629] node 0 deferred pages initialised in 200ms [ 0.271013] devtmpfs: initialized [ 0.272300] x86/mm: Memory block size: 128MB [ 0.275273] gcov: version magic: 0x41383552 [ 0.278156] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.281079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.283295] pinctrl core: initialized pinctrl subsystem [ 0.285167] [ 0.285719] ************************************************************* [ 0.288014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.290012] ** ** [ 0.292015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.295013] ** ** [ 0.297013] ** This means that this kernel is built to expose internal ** [ 0.299014] ** IOMMU data structures, which may compromise security on ** [ 0.302012] ** your system. ** [ 0.304009] ** ** [ 0.306015] ** If you see this message and you are not debugging the ** [ 0.308011] ** kernel, report this immediately to your vendor! ** [ 0.310021] ** ** [ 0.312012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.314012] ************************************************************* [ 0.317963] NET: Registered protocol family 16 [ 0.320599] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.324203] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.329065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.334193] cpuidle: using governor menu [ 0.335847] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.339667] PCI: Using configuration type 1 for base access [ 0.341125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.352068] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.355024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.360062] cryptd: max_cpu_qlen set to 1000 [ 0.364214] ACPI: Added _OSI(Module Device) [ 0.366017] ACPI: Added _OSI(Processor Device) [ 0.368013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.369013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.375753] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.382359] ACPI: Interpreter enabled [ 0.384062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.386013] ACPI: Using IOAPIC for interrupt routing [ 0.388099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.392433] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.404000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.408046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.411082] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.415081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.421553] acpiphp: Slot [2] registered [ 0.423114] acpiphp: Slot [5] registered [ 0.424193] acpiphp: Slot [6] registered [ 0.427139] acpiphp: Slot [7] registered [ 0.428310] acpiphp: Slot [8] registered [ 0.430126] acpiphp: Slot [9] registered [ 0.431099] acpiphp: Slot [10] registered [ 0.433117] acpiphp: Slot [3] registered [ 0.435112] acpiphp: Slot [4] registered [ 0.436093] acpiphp: Slot [11] registered [ 0.438163] acpiphp: Slot [12] registered [ 0.440025] acpiphp: Slot [13] registered [ 0.441112] acpiphp: Slot [14] registered [ 0.443120] acpiphp: Slot [15] registered [ 0.445105] acpiphp: Slot [16] registered [ 0.447121] acpiphp: Slot [17] registered [ 0.448107] acpiphp: Slot [18] registered [ 0.450096] acpiphp: Slot [19] registered [ 0.451212] acpiphp: Slot [20] registered [ 0.453119] acpiphp: Slot [21] registered [ 0.455122] acpiphp: Slot [22] registered [ 0.457129] acpiphp: Slot [23] registered [ 0.458096] acpiphp: Slot [24] registered [ 0.459000] acpiphp: Slot [25] registered [ 0.459000] acpiphp: Slot [26] registered [ 0.462128] acpiphp: Slot [27] registered [ 0.464089] acpiphp: Slot [28] registered [ 0.465096] acpiphp: Slot [29] registered [ 0.467121] acpiphp: Slot [30] registered [ 0.469080] acpiphp: Slot [31] registered [ 0.470068] PCI host bridge to bus 0000:00 [ 0.471018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.474024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.477025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.480023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.483024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.487025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.489167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.492750] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.496407] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.504882] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.511052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.513018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.516018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.518013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.520639] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.523951] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.526031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.529059] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.536016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.550018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.557016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.564836] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.571021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.580015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.609014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.624917] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.636014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.656016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.700018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.718481] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.727014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.736018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.753016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.764952] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.772015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.779016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.796013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.807634] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.829016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.839024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.861015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.874157] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.883122] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.894014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.913014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.924901] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.928376] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.931385] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.934396] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.936217] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.941655] iommu: Default domain type: Passthrough [ 0.943414] SCSI subsystem initialized [ 0.945127] ACPI: bus type USB registered [ 0.947138] usbcore: registered new interface driver usbfs [ 0.949081] usbcore: registered new interface driver hub [ 0.951094] usbcore: registered new device driver usb [ 0.953159] pps_core: LinuxPPS API ver. 1 registered [ 0.955012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.958049] PTP clock support registered [ 0.961065] EDAC MC: Ver: 3.0.0 [ 0.962194] PCI: Using ACPI for IRQ routing [ 0.964954] NetLabel: Initializing [ 0.966011] NetLabel: domain hash size = 128 [ 0.968014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.970083] NetLabel: unlabeled traffic allowed by default [ 0.972247] vgaarb: loaded [ 0.973271] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.975010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.980892] clocksource: Switched to clocksource kvm-clock [ 1.097790] VFS: Disk quotas dquot_6.6.0 [ 1.099425] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.102346] *** VALIDATE ramfs *** [ 1.103872] *** VALIDATE hugetlbfs *** [ 1.105878] pnp: PnP ACPI init [ 1.108593] pnp: PnP ACPI: found 6 devices [ 1.125894] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.128797] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.132229] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.135230] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.137908] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.140664] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.143426] NET: Registered protocol family 2 [ 1.145482] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.150191] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.154269] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.159956] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.163786] TCP: Hash tables configured (established 65536 bind 65536) [ 1.166744] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.170209] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.173406] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.176697] NET: Registered protocol family 1 [ 1.179984] RPC: Registered named UNIX socket transport module. [ 1.183064] RPC: Registered udp transport module. [ 1.184947] RPC: Registered tcp transport module. [ 1.186381] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.189240] NET: Registered protocol family 44 [ 1.191155] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.193518] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.196862] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.200256] PCI: CLS 0 bytes, default 64 [ 1.203317] Unpacking initramfs... [ 2.674165] debug: unmapping init [mem 0xffff9fc1bcc54000-0xffff9fc1bffbffff] [ 2.683199] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.687045] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.693069] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.173635] Initialise system trusted keyrings [ 3.175161] Key type blacklist registered [ 3.176939] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.185107] zbud: loaded [ 3.187983] *** VALIDATE nfs *** [ 3.189410] *** VALIDATE nfs4 *** [ 3.191069] pstore: using deflate compression [ 3.194267] Platform Keyring initialized [ 3.295473] NET: Registered protocol family 38 [ 3.297183] Key type asymmetric registered [ 3.298327] Asymmetric key parser 'x509' registered [ 3.299769] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.302471] io scheduler mq-deadline registered [ 3.304234] io scheduler kyber registered [ 3.305501] io scheduler bfq registered [ 3.307324] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.310186] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.312936] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.316091] ACPI: Power Button [PWRF] [ 3.321142] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.328282] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.342889] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.349602] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.364357] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.392240] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.421050] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.425308] Non-volatile memory driver v1.3 [ 3.426983] Linux agpgart interface v0.103 [ 3.465439] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.469187] vda: detected capacity change from 0 to 74698752 [ 3.487305] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.490942] vdb: detected capacity change from 0 to 1073741824 [ 3.510256] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.513068] vdc: detected capacity change from 0 to 2621440000 [ 3.530198] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.533263] vdd: detected capacity change from 0 to 2621440000 [ 3.551700] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.555624] vde: detected capacity change from 0 to 4294967296 [ 3.571752] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.575537] vdf: detected capacity change from 0 to 4294967296 [ 3.581813] libphy: Fixed MDIO Bus: probed [ 3.586687] usbcore: registered new interface driver usbserial_generic [ 3.589495] usbserial: USB Serial support registered for generic [ 3.591994] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.596835] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.598908] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.601306] mousedev: PS/2 mouse device common for all mice [ 3.603452] rtc_cmos 00:05: RTC can wake from S4 [ 3.605618] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.608854] rtc_cmos 00:05: registered as rtc0 [ 3.611608] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.611880] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.617075] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.618338] intel_pstate: CPU model not supported [ 3.623892] hid: raw HID events driver (C) Jiri Kosina [ 3.625401] usbcore: registered new interface driver usbhid [ 3.627037] usbhid: USB HID core driver [ 3.628327] drop_monitor: Initializing network drop monitor service [ 3.630629] Initializing XFRM netlink socket [ 3.632369] NET: Registered protocol family 10 [ 3.635043] Segment Routing with IPv6 [ 3.636340] NET: Registered protocol family 17 [ 3.639119] mpls_gso: MPLS GSO support [ 3.645863] RAS: Correctable Errors collector initialized. [ 3.647586] AVX version of gcm_enc/dec engaged. [ 3.648860] AES CTR mode by8 optimization enabled [ 3.723863] sched_clock: Marking stable (3723840952, 0)->(4658649696, -934808744) [ 3.727838] registered taskstats version 1 [ 3.729987] Loading compiled-in X.509 certificates [ 3.731932] zswap: loaded using pool lzo/zbud [ 3.757671] Key type big_key registered [ 3.772778] Key type encrypted registered [ 3.774217] ima: No TPM chip found, activating TPM-bypass! [ 3.776287] ima: Allocated hash algorithm: sha1 [ 3.777847] ima: No architecture policies found [ 3.779469] evm: Initialising EVM extended attributes: [ 3.780897] evm: security.selinux [ 3.782086] evm: security.ima [ 3.782846] evm: security.capability [ 3.783878] evm: HMAC attrs: 0x1 [ 3.785927] rtc_cmos 00:05: setting system clock to 2026-09-03 13:12:04 UTC (1788441124) [ 3.792036] debug: unmapping init [mem 0xffffffffbee03000-0xffffffffbeffffff] [ 3.795242] debug: unmapping init [mem 0xffffffffbdb82000-0xffffffffbde58fff] [ 3.804098] Write protecting the kernel read-only data: 28672k [ 3.808295] debug: unmapping init [mem 0xffffffffbc203000-0xffffffffbc3fffff] [ 3.811504] debug: unmapping init [mem 0xffffffffbcb14000-0xffffffffbcbfffff] [ 3.847919] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.856642] systemd[1]: Detected virtualization kvm. [ 3.858669] systemd[1]: Detected architecture x86-64. [ 3.860713] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.888657] systemd[1]: No hostname configured. [ 3.890701] systemd[1]: Set hostname to . [ 3.892819] random: systemd: uninitialized urandom read (16 bytes read) [ 3.895640] systemd[1]: Initializing machine ID from random generator. [ 3.942406] random: ln: uninitialized urandom read (6 bytes read) [ 4.036798] random: systemd: uninitialized urandom read (16 bytes read) [ 4.039989] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.044978] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.050371] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.769174] device-mapper: uevent: version 1.0.3 [ 4.771448] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.616990] virtio_net virtio0 ens2: renamed from eth0 [ 5.639676] random: fast init done [ 5.650311] scsi host0: ata_piix [ 5.697303] scsi host1: ata_piix [ 5.699117] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.702520] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.668851] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.289915] random: crng init done [ 10.291279] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.995913] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.270503] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.566402] SELinux: Disabled at runtime. [ 12.626406] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.634222] systemd[1]: Detected virtualization kvm. [ 12.636085] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.169150] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.173165] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.178110] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.182455] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.186484] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.195366] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.200113] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 13.263585] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.756166] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.103291] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.120637] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.372487] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.387349] EDAC sbridge: Ver: 1.1.2 [ 16.253783] Key type dns_resolver registered [ 16.601455] NFS: Registering the id_resolver key type [ 16.603370] Key type id_resolver registered [ 16.604843] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg315-server login: [ 44.425200] hrtimer: interrupt took 5125214 ns [ 66.388688] libcfs: loading out-of-tree module taints kernel. [ 66.431543] Key type ._llcrypt registered [ 66.433152] Key type .llcrypt registered [ 66.520407] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 90.187480] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 92.219720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 92.238721] alg: No test for adler32 (adler32-zlib) [ 93.805256] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 95.035665] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 96.920904] Key type lgssc registered [ 99.427864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 122.773891] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 166.346958] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 179.391415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 179.471411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.796490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 180.862445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 180.993662] Lustre: lustre-MDT0000: new disk, initializing [ 181.109204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.132957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 185.379528] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.772492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.889980] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 198.917807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 198.922886] Lustre: Skipped 1 previous similar message [ 199.007574] Lustre: lustre-MDT0001: new disk, initializing [ 199.082282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.123541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 199.157203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 203.035658] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.837933] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 217.795275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 218.107822] Lustre: lustre-OST0000: new disk, initializing [ 218.116090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 218.125809] Lustre: 8453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 218.220249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.043587] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.849651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 225.859205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 225.946830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 240.076022] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.228693] Lustre: lustre-OST0001: new disk, initializing [ 240.232825] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 240.240796] Lustre: 9526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.314300] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.341699] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.856144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 246.870670] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 246.969313] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 257.613461] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 267.423134] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 273.685110] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing check_logdir /tmp/testlogs/ [ 278.687880] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing yml_node [ 282.957561] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 285.607725] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 287.985225] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 289.847918] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:16:48 EDT 2026 [ 309.523276] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 311.385950] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 329.185500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.190291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 329.212033] Lustre: Skipped 1 previous similar message [ 334.180166] Lustre: server umount lustre-MDT0000 complete [ 337.678864] LustreError: 7462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441458 with bad export cookie 8348752744315606301 [ 337.681925] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 337.694424] LustreError: 7462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 339.433806] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.448558] LustreError: 10178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.449699] Lustre: Skipped 3 previous similar messages [ 339.453726] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 339.483210] LustreError: 10178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 339.508621] Lustre: Skipped 4 previous similar messages [ 342.433717] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.436696] LustreError: 10178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.474542] LustreError: 10178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 344.068040] Lustre: server umount lustre-MDT0001 complete [ 347.744199] Lustre: server umount lustre-OST0000 complete [ 351.376320] Lustre: server umount lustre-OST0001 complete [ 358.038635] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 367.167719] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 404.259601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.535852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 404.574997] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 404.680603] Lustre: lustre-MDT0000: new disk, initializing [ 404.810879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 404.849317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 410.099953] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 420.926727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.078291] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 421.127276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 421.131266] Lustre: Skipped 1 previous similar message [ 421.245390] Lustre: lustre-MDT0001: new disk, initializing [ 421.353045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 421.439462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 421.452861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 425.525657] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 434.458593] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 440.696696] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 447.124144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 447.374669] Lustre: lustre-OST0000: new disk, initializing [ 447.381106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 447.387406] Lustre: 17044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 447.488129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.523038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 448.532634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 448.680715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 453.229378] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 462.502081] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 466.962626] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 467.201332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 471.493796] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 471.671047] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 474.087269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 474.097474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.129960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 474.150098] Lustre: Skipped 1 previous similar message [ 474.599992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 479.601123] Lustre: server umount lustre-OST0000 complete [ 484.841543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.855577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 484.855586] Lustre: Skipped 1 previous similar message [ 484.870255] Lustre: Skipped 2 previous similar messages [ 489.808847] Lustre: server umount lustre-MDT0000 complete [ 489.969382] LustreError: 15087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.989597] LustreError: 15087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 494.040668] LustreError: 15873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441614 with bad export cookie 8348752744315607834 [ 494.052360] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 494.061065] LustreError: 15873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 494.608679] Lustre: server umount lustre-MDT0001 complete [ 502.129957] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:20:20 (1788441620) [ 523.028562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.753880] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.876708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 528.865741] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.934297] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 539.107731] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.131301] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 541.118220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.085542] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 554.522817] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 559.408501] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 561.157684] Lustre: Failing over lustre-MDT0000 [ 561.402891] Lustre: server umount lustre-MDT0000 complete [ 562.146062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.152797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 562.172809] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.569621] Lustre: 3649:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441682/real 1788441682] req@ffff9fc103c46a00 x1875316535615872/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788441698 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 577.595709] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.702735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.749018] LustreError: 3645:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fc237ad7480 x1875316535618944/t0(0) o250->MGC192.168.203.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 588.022588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 588.031805] Lustre: Skipped 1 previous similar message [ 592.745900] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 593.385532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 593.385573] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 593.416316] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 604.750485] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 612.131611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 612.133237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 612.140887] Lustre: Skipped 2 previous similar messages [ 612.150190] Lustre: Skipped 2 previous similar messages [ 615.338519] Lustre: server umount lustre-MDT0000 complete [ 617.449430] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.485499] LustreError: 19915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 619.486757] LustreError: 19899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441740 with bad export cookie 8348752744315609248 [ 619.509275] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 619.518614] LustreError: 19899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 619.965943] Lustre: server umount lustre-MDT0001 complete [ 629.423207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.324756] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 645.567263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.718996] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 658.770330] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 664.900152] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 666.938463] Lustre: Failing over lustre-MDT0000 [ 666.968000] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.989432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 667.265699] Lustre: server umount lustre-MDT0000 complete [ 686.560160] Lustre: 3648:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441791/real 1788441791] req@ffff9fc237ad2680 x1875316535648128/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788441807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 686.564613] LustreError: 23258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 686.597655] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 686.642388] LustreError: 23258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 687.492851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.841528] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x73dcba632a7fe44a [ 697.862354] Lustre: MGC192.168.203.115@tcp: Connection restored to 0@lo (at 0@lo) [ 697.872256] Lustre: Skipped 1 previous similar message [ 698.284607] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 703.462618] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 703.486580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 703.540551] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 704.713677] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 716.880374] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 724.450571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 724.467355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 724.481167] Lustre: Skipped 1 previous similar message [ 724.497826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 727.945931] Lustre: server umount lustre-MDT0000 complete [ 734.294667] LustreError: 23238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441855 with bad export cookie 8348752744315610186 [ 734.297616] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 734.322757] LustreError: 23238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 734.848094] Lustre: server umount lustre-MDT0001 complete [ 746.646656] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:24:25 (1788441865) [ 762.436557] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 774.686927] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 817.829935] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 828.600202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 828.814313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 828.853644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 828.963729] Lustre: lustre-MDT0000: new disk, initializing [ 829.028065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 829.032735] Lustre: Skipped 3 previous similar messages [ 829.042699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 833.323097] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 844.104531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.251327] Lustre: 29633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 844.451517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 848.913610] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 858.573619] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 864.934743] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 872.030237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 872.269521] Lustre: 31601:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 873.961839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 874.008113] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 874.051344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 874.070996] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 878.428953] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 888.337328] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 894.183087] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 920.038508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.043803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 920.052206] Lustre: Skipped 1 previous similar message [ 920.054383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 920.063454] LustreError: Skipped 1 previous similar message [ 920.085974] Lustre: Skipped 3 previous similar messages [ 922.881259] Lustre: server umount lustre-OST0000 complete [ 933.426457] Lustre: server umount lustre-MDT0000 complete [ 935.406174] LustreError: 29645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.425942] LustreError: 29645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 937.276827] LustreError: 29626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442057 with bad export cookie 8348752744315610599 [ 937.277469] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 937.288818] LustreError: 29626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 937.833503] Lustre: server umount lustre-MDT0001 complete [ 950.331180] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 953.638664] Key type lgssc unregistered [ 953.959535] LNet: 33584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.970548] LNetError: 33584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.000218] LNet: Removed LNI 192.168.203.115@tcp [ 954.986161] Key type .llcrypt unregistered [ 954.988546] Key type ._llcrypt unregistered [ 980.968924] Key type ._llcrypt registered [ 980.973649] Key type .llcrypt registered [ 981.118088] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 982.248447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 982.320453] alg: No test for adler32 (adler32-zlib) [ 983.469305] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 983.731927] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 985.472763] Key type lgssc registered [ 986.436630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 993.892574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 993.913777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 995.328393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 999.908090] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1004.515411] LustreError: 34998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.535836] LustreError: 34998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1007.908600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.351098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1012.998453] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1021.019753] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1027.124429] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1034.004991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1034.326628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.490566] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1040.772967] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1048.690494] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1052.834162] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1066.110981] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:29:44 (1788442184) [ 1068.135221] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1069.999162] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1078.546339] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:29:57 (1788442197) [ 1088.896589] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:30:07 (1788442207) [ 1090.096256] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1099.430139] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1099.442547] Lustre: Skipped 2 previous similar messages [ 1101.414558] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:30:20 (1788442220) [ 1103.622938] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1103.629904] Lustre: Skipped 2 previous similar messages [ 1111.959810] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1111.968714] Lustre: Skipped 4 previous similar messages [ 1127.266725] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:30:46 (1788442246) [ 1130.660262] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1134.455594] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1143.641530] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:31:02 (1788442262) [ 1145.934812] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1145.943478] Lustre: Skipped 6 previous similar messages [ 1160.585341] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:31:19 (1788442279) [ 1176.037611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.039663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1176.064828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1176.088408] Lustre: Skipped 1 previous similar message [ 1181.154342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1181.162810] Lustre: Skipped 1 previous similar message [ 1186.279670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1186.287968] Lustre: Skipped 1 previous similar message [ 1189.344377] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1189.528885] Lustre: server umount lustre-OST0000 complete [ 1196.513751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.518853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.532285] Lustre: Skipped 2 previous similar messages [ 1196.539915] Lustre: Skipped 1 previous similar message [ 1200.142381] Lustre: server umount lustre-MDT0000 complete [ 1201.634854] LustreError: 34995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.664150] LustreError: 34995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1204.943191] LustreError: 34979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442325 with bad export cookie 10652615526222649479 [ 1204.965354] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1205.479261] Lustre: server umount lustre-MDT0001 complete [ 1221.261582] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1224.890919] Key type lgssc unregistered [ 1225.250173] LNet: 42861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1225.262210] LNetError: 42861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1225.289231] LNet: Removed LNI 192.168.203.115@tcp [ 1226.117789] Key type .llcrypt unregistered [ 1226.120834] Key type ._llcrypt unregistered [ 1244.743863] Key type ._llcrypt registered [ 1244.749304] Key type .llcrypt registered [ 1244.877656] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 1261.388465] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1262.318296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1262.355069] alg: No test for adler32 (adler32-zlib) [ 1263.401490] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1263.653635] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1265.328352] Key type lgssc registered [ 1266.529317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1302.082540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1302.109346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.452660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1303.506527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1303.627478] Lustre: lustre-MDT0000: new disk, initializing [ 1303.744559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1303.767304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1308.587106] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1319.824992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.973133] Lustre: 46016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1320.047065] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1320.053422] Lustre: Skipped 1 previous similar message [ 1320.160102] Lustre: lustre-MDT0001: new disk, initializing [ 1320.278526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1320.338171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1320.352692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1324.881579] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1335.675801] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1342.448964] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1350.788692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1351.159690] Lustre: lustre-OST0000: new disk, initializing [ 1351.163610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1351.171201] Lustre: 47986:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1351.252671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1357.419732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1357.456673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1357.572079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1358.155748] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1369.209864] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1373.802336] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1373.967955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1378.037256] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1378.271733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1383.398959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.404499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1383.406455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1383.423475] Lustre: Skipped 1 previous similar message [ 1383.455451] Lustre: Skipped 1 previous similar message [ 1386.388783] Lustre: server umount lustre-OST0000 complete [ 1392.101119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1392.109471] LustreError: Skipped 1 previous similar message [ 1392.114361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.125866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1393.634827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1393.639248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.129608] Lustre: server umount lustre-MDT0000 complete [ 1401.136181] LustreError: 46008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442521 with bad export cookie 4997711838453449119 [ 1401.139979] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1401.159473] LustreError: 46008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1401.636282] Lustre: server umount lustre-MDT0001 complete [ 1412.924316] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:35:31 (1788442531) [ 1420.470759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.976313] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.993454] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1421.036791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1426.256811] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1426.406397] LustreError: 50448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.426753] LustreError: 50448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1430.509347] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1435.434584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1435.619254] LustreError: 50448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1435.942619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1441.022462] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1449.020504] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1454.764150] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1461.608990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1462.016355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.592702] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1476.785249] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1485.078986] LustreError: 52976:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1486.418064] Lustre: Modifying parameter general.timeout=1 in log params [ 1487.480697] Lustre: Modifying parameter general.timeout=2 in log params [ 1489.629644] Lustre: Modifying parameter general.timeout=4 in log params [ 1489.638161] Lustre: Skipped 1 previous similar message [ 1494.529890] Lustre: Modifying parameter general.timeout=9 in log params [ 1494.539149] Lustre: Skipped 4 previous similar messages [ 1503.622207] Lustre: Modifying parameter general.timeout=17 in log params [ 1503.628573] Lustre: Skipped 7 previous similar messages [ 1519.970682] Lustre: Modifying parameter general.timeout=34 in log params [ 1519.976937] Lustre: Skipped 16 previous similar messages [ 1552.796249] Lustre: Modifying parameter general.timeout=69 in log params [ 1552.808575] Lustre: Skipped 34 previous similar messages [ 1617.422609] Lustre: Disabling parameter general.timeout= in log params [ 1617.431054] Lustre: Skipped 60 previous similar messages [ 1620.564953] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:38:59 (1788442739) [ 1623.619630] Lustre: 59919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1625.265040] Lustre: 60018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1629.330080] Lustre: 60231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1629.347775] Lustre: 60231:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1662.135116] Lustre: 61797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1662.149650] Lustre: 61797:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1662.176274] LustreError: 61797:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1666.378813] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:39:45 (1788442785) [ 1681.891847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1681.902410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.915714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1681.920326] Lustre: Skipped 1 previous similar message [ 1684.762118] Lustre: server umount lustre-MDT0000 complete [ 1688.036956] LustreError: 50448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1688.061654] LustreError: 50448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1688.843667] LustreError: 52290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442809 with bad export cookie 4997711838453450001 [ 1688.855747] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1688.856448] LustreError: 52290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1689.323766] Lustre: server umount lustre-MDT0001 complete [ 1703.941791] Lustre: server umount lustre-OST0000 complete [ 1732.596556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1732.985601] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1733.353667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.329198] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1747.987622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1748.149204] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1748.197774] Lustre: 64327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1754.770822] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1763.267865] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1770.679205] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1777.336116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1777.546821] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1777.781990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1777.805204] Lustre: Skipped 1 previous similar message [ 1784.760089] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1794.607590] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1796.730967] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1796.739475] Lustre: Skipped 23 previous similar messages [ 1796.786357] Lustre: 66897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1810.917368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1810.930039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.940428] Lustre: Skipped 2 previous similar messages [ 1810.948325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1810.956824] Lustre: Skipped 3 previous similar messages [ 1813.352952] Lustre: server umount lustre-OST0000 complete [ 1820.131058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1820.141412] LustreError: Skipped 1 previous similar message [ 1820.147474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1820.167629] Lustre: Skipped 1 previous similar message [ 1820.180312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1820.187705] Lustre: Skipped 2 previous similar messages [ 1823.508618] Lustre: server umount lustre-MDT0000 complete [ 1826.785240] LustreError: 64340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.821080] LustreError: 64340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1827.537948] LustreError: 64321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442948 with bad export cookie 4997711838453453025 [ 1827.546547] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.575940] LustreError: 64321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1827.885802] Lustre: server umount lustre-MDT0001 complete [ 1838.056686] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1840.950600] Key type lgssc unregistered [ 1841.163917] LNet: 68110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.170569] LNetError: 68110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.184272] LNet: Removed LNI 192.168.203.115@tcp [ 1842.037733] Key type .llcrypt unregistered [ 1842.041303] Key type ._llcrypt unregistered [ 1870.128320] Key type ._llcrypt registered [ 1870.129687] Key type .llcrypt registered [ 1870.234745] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1871.447778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1871.471713] alg: No test for adler32 (adler32-zlib) [ 1872.574338] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1872.832936] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1874.456513] Key type lgssc registered [ 1875.526730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1883.339627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1883.363887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.876593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.586897] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1894.381632] LustreError: 69524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.409575] LustreError: 69524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1899.117272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.835188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1904.308162] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1913.396836] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1921.368217] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1929.209858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1930.017313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.606907] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1946.997190] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1955.332081] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1965.030213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.045715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1965.046714] Lustre: Skipped 1 previous similar message [ 1965.066587] Lustre: Skipped 1 previous similar message [ 1970.150553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1970.168237] Lustre: Skipped 2 previous similar messages [ 1970.539031] Lustre: server umount lustre-MDT0000 complete [ 1974.803959] LustreError: 69504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443095 with bad export cookie 3882347882340833259 [ 1974.814725] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1974.819432] LustreError: 69504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1975.343148] Lustre: server umount lustre-MDT0001 complete [ 1989.935864] Lustre: server umount lustre-OST0000 complete [ 2017.304908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.668241] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2017.732771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2018.014421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.154343] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2030.956070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.053236] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2031.076552] Lustre: 74398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2031.721256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2036.791490] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2045.774729] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2051.982629] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2060.574657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2060.705616] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2061.189091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.459521] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2077.382418] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2078.928734] Lustre: 76973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2080.852258] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2086.372309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.385439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2086.389113] Lustre: Skipped 2 previous similar messages [ 2086.408061] Lustre: Skipped 1 previous similar message [ 2091.493456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2091.499294] Lustre: Skipped 1 previous similar message [ 2092.279632] Lustre: server umount lustre-OST0000 complete [ 2096.611390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.613793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2096.623382] Lustre: Skipped 1 previous similar message [ 2096.632319] Lustre: Skipped 1 previous similar message [ 2102.704489] Lustre: server umount lustre-MDT0000 complete [ 2106.371600] LustreError: 74390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443227 with bad export cookie 3882347882340834085 [ 2106.377923] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2106.386670] LustreError: 74390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2106.826277] Lustre: server umount lustre-MDT0001 complete [ 2116.486900] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2118.927236] Key type lgssc unregistered [ 2119.229924] LNet: 78186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2119.239274] LNetError: 78186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.294951] LNet: Removed LNI 192.168.203.115@tcp [ 2121.103352] Key type .llcrypt unregistered [ 2121.107321] Key type ._llcrypt unregistered [ 2145.881325] Key type ._llcrypt registered [ 2145.885056] Key type .llcrypt registered [ 2145.999286] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2146.891460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2146.948413] alg: No test for adler32 (adler32-zlib) [ 2147.978917] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2148.271287] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2149.928141] Key type lgssc registered [ 2151.018753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2158.714487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2158.772268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.287562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2164.716343] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2169.838920] LustreError: 79600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.857845] LustreError: 79600:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2172.650927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.228627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2177.727947] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2185.542302] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2192.636680] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2200.432832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2200.921186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.471814] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2214.161235] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2224.143772] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2225.156992] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2227.176650] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2227.186500] Lustre: Skipped 1 previous similar message [ 2229.763240] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2229.770161] Lustre: Skipped 2 previous similar messages [ 2234.444880] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2234.465431] Lustre: Skipped 4 previous similar messages [ 2242.730763] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2242.740540] Lustre: Skipped 8 previous similar messages [ 2259.286348] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2259.294053] Lustre: Skipped 15 previous similar messages [ 2291.550214] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2291.565695] Lustre: Skipped 37 previous similar messages [ 2320.003506] Lustre: 87703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2322.843939] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:50:41 (1788443441) [ 2324.411395] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2326.179496] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:50:45 (1788443445) [ 2330.082384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2330.094714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2330.106245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2330.109622] Lustre: Skipped 1 previous similar message [ 2335.203674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2335.210468] Lustre: Skipped 2 previous similar messages [ 2335.918783] Lustre: server umount lustre-OST0000 complete [ 2340.321330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.334793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2340.341828] Lustre: Skipped 1 previous similar message [ 2340.348406] Lustre: Skipped 1 previous similar message [ 2345.442358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2345.451147] Lustre: Skipped 1 previous similar message [ 2345.851940] Lustre: server umount lustre-MDT0000 complete [ 2349.848941] LustreError: 80344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443470 with bad export cookie 5128026116762439148 [ 2349.856116] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2349.864069] LustreError: 80344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2350.308432] Lustre: server umount lustre-MDT0001 complete [ 2362.076031] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2365.455761] Key type lgssc unregistered [ 2365.836713] LNet: 88956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2365.843845] LNetError: 88956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2365.860255] LNet: Removed LNI 192.168.203.115@tcp [ 2366.730891] Key type .llcrypt unregistered [ 2366.733071] Key type ._llcrypt unregistered [ 2370.661888] Key type ._llcrypt registered [ 2370.667632] Key type .llcrypt registered [ 2370.842926] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2375.531394] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules [ 2376.996250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2377.048840] alg: No test for adler32 (adler32-zlib) [ 2378.221633] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2378.549051] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2380.320250] Key type lgssc registered [ 2381.780693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2382.534611] LustreError: 90158:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2398.093378] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2404.380621] LustreError: 90158:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2408.068697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2408.129840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2409.741833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2415.115594] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2420.196840] LustreError: 91182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.210750] LustreError: 91182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2425.313748] LustreError: 91183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.020662] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:52:25 (1788443545) [ 2430.439031] LustreError: 91182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.845014] LustreError: 91214:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2431.859170] LustreError: 91214:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2431.864649] LustreError: 91214:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2432.139831] Lustre: server umount lustre-MDT0000 complete [ 2445.907709] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2448.796871] Key type lgssc unregistered [ 2449.075333] LNet: 92710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2449.081865] LNetError: 92710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2450.155973] LNet: Removed LNI 192.168.203.115@tcp [ 2450.927345] Key type .llcrypt unregistered [ 2450.932134] Key type ._llcrypt unregistered [ 2475.740910] Key type ._llcrypt registered [ 2475.747807] Key type .llcrypt registered [ 2475.950551] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2477.732543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2477.810114] alg: No test for adler32 (adler32-zlib) [ 2478.862844] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2479.096103] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2480.832212] Key type lgssc registered [ 2482.395419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2490.923938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2490.943715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.581724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2496.882260] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2503.140701] LustreError: 93930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.157985] LustreError: 93930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2508.208093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.271267] LustreError: 93931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.430946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2514.414643] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2522.611713] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2529.677878] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2538.284111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2539.205790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.252615] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2556.009971] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2623.216991] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:55:42 (1788443742) [ 2625.226663] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2626.891916] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:55:45 (1788443745) [ 2631.652554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2631.657954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.685269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2632.673354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2632.687053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2632.727323] Lustre: Skipped 1 previous similar message [ 2637.631738] Lustre: server umount lustre-MDT0000 complete [ 2637.795279] LustreError: 94674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.811838] LustreError: 94674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2641.664410] LustreError: 93910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443762 with bad export cookie 1679592779178583021 [ 2641.669879] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2641.672112] LustreError: 93910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2642.239752] Lustre: server umount lustre-MDT0001 complete [ 2658.287125] Lustre: server umount lustre-OST0000 complete [ 2671.392361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.829563] LustreError: 98257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.041210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.222523] LustreError: 98258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.251642] LustreError: 98258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2677.663547] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2686.434217] LustreError: 98258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.459988] LustreError: 98258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2686.596022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.381445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2692.131220] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2699.299914] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2704.731954] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2721.451672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2721.674190] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2721.687271] LustreError: 98245:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2721.715601] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2721.723792] LustreError: 100339:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2721.728818] LustreError: 100339:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2721.738778] LustreError: 100339:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2721.848946] Lustre: server umount lustre-OST0000 complete [ 2721.852700] LustreError: 100339:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2731.134665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2731.294128] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2731.305845] LustreError: 98245:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2731.323799] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2731.345239] LustreError: 100808:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2731.356562] LustreError: 100808:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2731.368492] LustreError: 100808:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2731.427407] Lustre: server umount lustre-OST0000 complete [ 2731.429761] LustreError: 100808:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2743.088842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2743.415065] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2743.429131] Lustre: Found index 0 for lustre-OST0000, updating log [ 2743.438741] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2743.551311] Lustre: lustre-OST0000: new disk, initializing [ 2743.558536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2743.569618] Lustre: 101344:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2743.963103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2750.163337] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2753.840507] Lustre: Failing over lustre-OST0000 [ 2753.981180] Lustre: server umount lustre-OST0000 complete [ 2758.629889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2758.645494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.658633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2758.671873] Lustre: Skipped 2 previous similar messages [ 2762.209952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.211373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.241241] Lustre: Skipped 1 previous similar message [ 2764.756887] Lustre: server umount lustre-MDT0000 complete [ 2767.331262] LustreError: 98258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.356718] LustreError: 98258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2768.685471] LustreError: 100358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443889 with bad export cookie 1679592779178585758 [ 2768.692074] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2768.697676] LustreError: 100358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2769.090234] Lustre: server umount lustre-MDT0001 complete [ 2778.619028] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:58:17 (1788443897) [ 2796.402705] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2807.195272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.594291] LustreError: 103687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2807.702845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2812.398898] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2822.218318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2827.617901] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2831.290531] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2839.667709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2840.284232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.296153] Lustre: Skipped 1 previous similar message [ 2848.786772] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2859.895343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2860.058646] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2860.160768] Lustre: lustre-OST0001: new disk, initializing [ 2860.164225] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2860.172139] Lustre: 106005:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2868.843042] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2869.382972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2869.415377] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2869.544168] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2878.048980] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2885.250911] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2892.300206] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 2894.819227] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2894.830232] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.955357] Lustre: server umount lustre-MDT0001 complete [ 2897.329011] LustreError: 103683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2897.364385] LustreError: 103683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2913.343258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2913.447804] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2913.457709] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2913.521881] Lustre: 103677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2915.213817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2915.228516] Lustre: Skipped 1 previous similar message [ 2915.529443] LustreError: 107530:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2915.556751] LustreError: 107530:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2915.589844] LustreError: 103679:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2915.742526] LustreError: 107543:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.115@tcp: cfg command failed: rc = -17 [ 2915.750458] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2915.750458] [ 2920.468292] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2923.262721] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2933.950868] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:00:52 (1788444052) [ 2953.832731] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2973.274054] Lustre: 109636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3002.733566] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3008.880648] Lustre: 110773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3108.837164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3108.860222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3108.893112] Lustre: Skipped 2 previous similar messages [ 3108.901708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3108.912907] Lustre: Skipped 4 previous similar messages [ 3114.143274] Lustre: server umount lustre-MDT0000 complete [ 3114.982161] LustreError: 103687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3115.007139] LustreError: 103687:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 3118.676845] LustreError: 104429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444239 with bad export cookie 1679592779178586759 [ 3118.683858] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3118.685499] LustreError: 104429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3119.145421] Lustre: server umount lustre-MDT0001 complete [ 3133.707687] Lustre: server umount lustre-OST0000 complete [ 3136.290965] Lustre: 93242:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444240/real 1788444240] req@ffff9fc230327b80 x1875319035659520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788444256 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3136.318072] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3136.336685] Lustre: Skipped 3 previous similar messages [ 3137.421302] Lustre: server umount lustre-OST0001 complete [ 3143.399362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3153.558672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3162.357542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3173.106625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3197.841084] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3209.724197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3209.750227] Lustre: lustre-MDT0000: reset Object Index mappings [ 3210.396350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3215.912214] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3227.082956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3227.100768] Lustre: lustre-MDT0001: reset Object Index mappings [ 3234.010245] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3237.325846] Lustre: 115877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3245.405107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3252.790793] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3253.511586] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3255.862585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3263.345836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3269.629229] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3269.652649] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3271.930302] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3280.882063] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3286.557139] Lustre: 117752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3345.109373] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:07:43 (1788444463) [ 3372.003038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3372.036079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3372.061284] Lustre: Skipped 2 previous similar messages [ 3372.088437] Lustre: Skipped 7 previous similar messages [ 3377.786964] Lustre: server umount lustre-MDT0000 complete [ 3380.205705] LustreError: 115791:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3380.247291] LustreError: 115791:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 3382.135785] LustreError: 114710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444502 with bad export cookie 1679592779178633624 [ 3382.146067] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3382.153019] LustreError: 114710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3388.927713] Lustre: server umount lustre-MDT0001 complete [ 3399.882953] Lustre: server umount lustre-OST0000 complete [ 3404.571758] LustreError: 118719:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0xbd:0x0].0x0 (ffff9fc23c074200) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3410.900487] Lustre: server umount lustre-OST0001 complete [ 3418.193446] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 3430.028409] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3472.804146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.167513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3473.206876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3473.297314] Lustre: lustre-MDT0000: new disk, initializing [ 3473.364195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.368352] Lustre: Skipped 3 previous similar messages [ 3473.379780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3478.261359] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3485.242706] Lustre: server umount lustre-MDT0000 complete [ 3495.520072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3495.830638] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3495.852424] Lustre: 122029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3495.873238] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3500.569640] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3505.775381] Lustre: server umount lustre-MDT0000 complete [ 3515.624870] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:10:34 (1788444634) [ 3517.328424] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3519.473514] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:10:38 (1788444638) [ 3540.464797] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3551.094730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.205897] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3565.083108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3565.170810] Lustre: 124081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3565.215937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3565.218330] Lustre: Skipped 1 previous similar message [ 3565.271958] Lustre: lustre-MDT0001: new disk, initializing [ 3565.432871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3565.441619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3570.084844] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3574.872171] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3581.705565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3581.969091] Lustre: lustre-OST0000: new disk, initializing [ 3581.974754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3581.983599] Lustre: 125656:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3583.531088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3583.541440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3583.626432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3588.581631] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3599.282494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3599.497128] Lustre: lustre-OST0001: new disk, initializing [ 3599.501787] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3599.509354] Lustre: 126527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3600.788633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3600.807374] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3600.900743] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3606.087222] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3618.077275] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3622.888268] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4097.945726] Lustre: 132957:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4097.965793] Lustre: 132957:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4097.972849] Lustre: 132957:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4097.978195] Lustre: 132957:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4097.992709] Lustre: 132957:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4098.005814] Lustre: 132957:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4431.718359] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:25:49 (1788445549) [ 4450.796228] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.803681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4450.809457] Lustre: Skipped 6 previous similar messages [ 4450.812339] Lustre: Skipped 10 previous similar messages [ 4453.474882] Lustre: server umount lustre-MDT0000 complete [ 4455.915521] LustreError: 127334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4455.955779] LustreError: 127334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4458.656186] LustreError: 124071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445579 with bad export cookie 1679592779178703827 [ 4458.664062] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4458.678065] LustreError: 124071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4461.030664] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4461.056149] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4461.058738] Lustre: Skipped 1 previous similar message [ 4465.267299] Lustre: server umount lustre-MDT0001 complete [ 4476.933934] Lustre: server umount lustre-OST0000 complete [ 4496.865860] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4497.286269] Lustre: server umount lustre-OST0001 complete [ 4504.776866] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 4516.371776] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4556.190589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4556.639524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4556.703379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4556.839722] Lustre: lustre-MDT0000: new disk, initializing [ 4557.055028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4557.074852] Lustre: Skipped 5 previous similar messages [ 4557.126201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4562.377600] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4573.312404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4573.502715] Lustre: 136024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4573.560590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4573.565483] Lustre: Skipped 1 previous similar message [ 4573.671677] Lustre: lustre-MDT0001: new disk, initializing [ 4573.842329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4573.858928] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4578.280207] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4589.452392] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4597.190936] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4605.358978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4605.790326] Lustre: lustre-OST0000: new disk, initializing [ 4605.794575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4605.803402] Lustre: 137994:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4607.178079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4607.197296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4607.345421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4612.723738] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4622.676206] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4633.587863] Lustre: lustre-MDD0000: changelog on [ 4636.593401] Lustre: lustre-MDD0001: changelog on [ 6704.068518] Lustre: lustre-MDD0001: changelog off [ 6705.382436] Lustre: 137175:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6706.103828] Lustre: lustre-MDD0000: changelog off [ 6711.783103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6711.786309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6711.790770] LustreError: Skipped 1 previous similar message [ 6711.797277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6711.802169] Lustre: Skipped 2 previous similar messages [ 6711.816720] Lustre: Skipped 3 previous similar messages [ 6716.909773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6716.930603] Lustre: Skipped 1 previous similar message [ 6717.300387] Lustre: server umount lustre-OST0000 complete [ 6722.028354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6722.047751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6722.049667] Lustre: Skipped 1 previous similar message [ 6722.066191] Lustre: Skipped 1 previous similar message [ 6727.141384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6727.156273] Lustre: Skipped 1 previous similar message [ 6728.208397] Lustre: server umount lustre-MDT0000 complete [ 6732.186913] LustreError: 136017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447852 with bad export cookie 1679592779179526586 [ 6732.187617] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6732.200912] LustreError: 136017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6732.669811] Lustre: server umount lustre-MDT0001 complete [ 6743.666355] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6746.595962] Key type lgssc unregistered [ 6746.877074] LNet: 147175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6746.884665] LNetError: 147175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6746.919635] LNet: Removed LNI 192.168.203.115@tcp [ 6747.955243] Key type .llcrypt unregistered [ 6747.962047] Key type ._llcrypt unregistered [ 6773.108612] Key type ._llcrypt registered [ 6773.111569] Key type .llcrypt registered [ 6773.281176] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 6792.322131] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6793.379791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6793.405348] alg: No test for adler32 (adler32-zlib) [ 6794.532947] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6794.857822] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 6796.592203] Key type lgssc registered [ 6797.638729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6831.309057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6831.332532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6832.783852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6832.852193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6832.988565] Lustre: lustre-MDT0000: new disk, initializing [ 6833.049874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6833.108812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6838.386103] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6849.905207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6850.073525] Lustre: 150378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6850.122399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6850.125931] Lustre: Skipped 1 previous similar message [ 6850.193778] Lustre: lustre-MDT0001: new disk, initializing [ 6850.304802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6850.341133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6850.353187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6854.853449] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6864.928585] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6871.966877] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6882.088274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6882.878372] Lustre: lustre-OST0000: new disk, initializing [ 6882.893406] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6882.906053] Lustre: 152349:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6883.166031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6889.094979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6889.129145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6889.298767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6892.268175] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6903.515591] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6910.594263] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_module obdecho/obdecho [ 6912.547298] LustreError: 153354:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6912.564700] LustreError: 153354:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6914.563109] LustreError: 153450:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6927.334682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6927.342078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6927.362207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6930.406204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6930.409602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6930.418342] Lustre: Skipped 1 previous similar message [ 6930.427308] Lustre: Skipped 2 previous similar messages [ 6931.624588] Lustre: server umount lustre-MDT0000 complete [ 6934.695710] LustreError: 150373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448055 with bad export cookie 12689695980842465112 [ 6934.702164] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6934.704885] LustreError: 150373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6934.974945] Lustre: server umount lustre-MDT0001 complete [ 6947.414068] Lustre: server umount lustre-OST0000 complete [ 6964.556642] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 6973.347137] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7014.803919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7015.134846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7015.172875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7015.250433] Lustre: lustre-MDT0000: new disk, initializing [ 7015.502542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7015.537754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7021.870411] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7034.729362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7034.794249] Lustre: 157165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7034.839953] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7034.843626] Lustre: Skipped 1 previous similar message [ 7034.893324] Lustre: lustre-MDT0001: new disk, initializing [ 7034.995559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7035.033099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7035.053363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7039.490735] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7050.524673] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7057.310353] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7065.670175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7066.020285] Lustre: lustre-OST0000: new disk, initializing [ 7066.026729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7066.036459] Lustre: 159135:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7066.132264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7072.631538] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7073.925577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7073.952194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7074.040737] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7083.677823] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7088.748127] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7088.994549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7093.918916] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7094.201279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7099.878739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7099.885232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7099.889095] LustreError: Skipped 1 previous similar message [ 7099.890719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7099.933906] Lustre: Skipped 1 previous similar message [ 7102.709503] Lustre: server umount lustre-OST0000 complete [ 7106.542088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7106.551481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7106.572203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7106.592232] Lustre: Skipped 1 previous similar message [ 7112.848464] Lustre: server umount lustre-MDT0000 complete [ 7116.780741] LustreError: 159134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448237 with bad export cookie 12689695980842466071 [ 7116.788835] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7117.336071] Lustre: server umount lustre-MDT0001 complete [ 7120.926919] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:10:39 (1788448239) [ 7122.590632] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7124.942865] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:10:43 (1788448243) [ 7133.763127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.313549] LustreError: 161549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7134.326623] LustreError: 161549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7134.391308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7139.824018] LustreError: 161550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7140.022459] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7144.932459] LustreError: 161549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7149.316913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7154.265619] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7162.465135] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7168.271501] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7175.157471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7175.445235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7175.457451] Lustre: Skipped 1 previous similar message [ 7182.081201] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7191.991823] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7201.250488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7201.261485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7201.294847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7201.305146] Lustre: Skipped 2 previous similar messages [ 7201.311587] Lustre: Skipped 4 previous similar messages [ 7205.337131] Lustre: server umount lustre-MDT0000 complete [ 7206.376286] LustreError: 162280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.408239] LustreError: 162280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7209.580511] LustreError: 161530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448330 with bad export cookie 12689695980842466953 [ 7209.596261] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7211.489809] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7211.490967] LustreError: 163455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7211.504169] Lustre: Skipped 1 previous similar message [ 7211.504913] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7211.504917] Lustre: Skipped 3 previous similar messages [ 7211.536383] LustreError: 163455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7216.193590] Lustre: server umount lustre-MDT0001 complete [ 7222.831485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7238.312264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7238.614302] LustreError: 165398:0:(llog.c:1655:llog_backup()) MGC192.168.203.115@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7238.622091] Lustre: 165398:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.115@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7240.161790] LustreError: 165403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7240.176956] LustreError: 165403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7259.620096] LustreError: 165405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7259.660525] LustreError: 165405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 7295.394655] LustreError: 165403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7295.421222] LustreError: 165403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 7300.064762] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7300.072913] Lustre: 165398:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7341.061795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7346.168597] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7346.430736] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7356.502389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7359.909154] LustreError: 165444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7359.950597] LustreError: 165444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 7371.757695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7381.993235] LustreError: 148600:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003adf9799 x1875323561057408/t0(0) o250->MGC192.168.203.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7387.656786] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7388.649557] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7397.266754] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7399.021888] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7405.825786] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7407.728161] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7428.578819] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7428.579490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7428.586937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7428.586946] Lustre: Skipped 2 previous similar messages [ 7428.599600] Lustre: Skipped 1 previous similar message [ 7432.893748] Lustre: server umount lustre-MDT0000 complete [ 7436.539729] LustreError: 161531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448557 with bad export cookie 12689695980842473449 [ 7436.544736] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7436.548470] LustreError: 161531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7436.939191] Lustre: server umount lustre-MDT0001 complete [ 7451.376188] Lustre: server umount lustre-OST0000 complete [ 7473.483743] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 7483.496395] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7519.728087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.990654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7520.038598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7520.149661] Lustre: lustre-MDT0000: new disk, initializing [ 7520.272952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7520.279214] Lustre: Skipped 1 previous similar message [ 7520.307733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7525.208418] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7535.548343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7535.660870] Lustre: 170509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7535.983596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7540.181731] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7549.309694] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7555.564126] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7562.107939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7562.468742] Lustre: lustre-OST0000: new disk, initializing [ 7562.471501] Lustre: Skipped 1 previous similar message [ 7562.477995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7562.484973] Lustre: Skipped 2 previous similar messages [ 7562.492665] Lustre: 172477:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7568.432375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7568.442385] Lustre: Skipped 1 previous similar message [ 7568.452229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7568.565850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7568.962371] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7578.750727] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7584.412680] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7584.607513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7589.413325] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7589.617909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7594.473038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7594.474367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7594.484671] LustreError: Skipped 1 previous similar message [ 7594.485593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7594.485600] Lustre: Skipped 2 previous similar messages [ 7594.527291] Lustre: Skipped 1 previous similar message [ 7606.242884] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7606.545404] Lustre: server umount lustre-OST0000 complete [ 7617.691430] Lustre: server umount lustre-MDT0000 complete [ 7620.071891] LustreError: 170520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7620.090612] LustreError: 170520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 7622.726292] LustreError: 171318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448743 with bad export cookie 12689695980842490256 [ 7622.737641] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7622.765240] LustreError: 171318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7623.198633] Lustre: server umount lustre-MDT0001 complete [ 7633.714667] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:19:12 (1788448752) [ 7641.083919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7646.258640] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7655.198820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.827568] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7665.990676] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7671.525977] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7677.527077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7683.629724] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7690.228446] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7700.194793] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:20:19 (1788448819) [ 7706.597263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7706.601680] LustreError: Skipped 1 previous similar message [ 7706.610674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7706.623267] Lustre: Skipped 2 previous similar messages [ 7706.628802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7706.635612] Lustre: Skipped 8 previous similar messages [ 7710.067651] Lustre: server umount lustre-OST0000 complete [ 7719.516471] Lustre: server umount lustre-MDT0000 complete [ 7722.950584] LustreError: 174868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448843 with bad export cookie 12689695980842491131 [ 7722.961282] LustreError: 174868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7722.961363] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7723.332937] Lustre: server umount lustre-MDT0001 complete [ 7733.635764] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7736.052886] Key type lgssc unregistered [ 7736.405891] LNet: 178619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7736.418037] LNetError: 178619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7736.446723] LNet: Removed LNI 192.168.203.115@tcp [ 7737.208397] Key type .llcrypt unregistered [ 7737.211068] Key type ._llcrypt unregistered [ 7753.751400] Key type ._llcrypt registered [ 7753.755582] Key type .llcrypt registered [ 7753.827223] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7754.408979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7754.423718] alg: No test for adler32 (adler32-zlib) [ 7755.492349] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7755.695566] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7757.368754] Key type lgssc registered [ 7758.469339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7765.292897] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7765.329302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7780.897340] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7786.017250] LustreError: 179878:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 7791.137568] LustreError: 179878:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7791.166493] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7791.179150] LustreError: MGC192.168.203.115@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7791.194036] LustreError: 179878:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7791.206325] LustreError: 179878:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7791.217920] LustreError: 179878:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7791.279831] Lustre: server umount lustre-OST0000 complete [ 7791.286415] LustreError: 179878:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7799.660254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7801.059630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7805.346720] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7810.531546] LustreError: 180376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7810.555652] LustreError: 180376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7814.431062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7814.772094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7819.194924] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7826.624479] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7831.857100] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7838.828330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7839.044430] LustreError: 182192:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7839.059697] Lustre: 182192:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.115@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7839.289467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7845.917561] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7854.345783] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7860.710629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7860.723906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7860.724399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7860.738509] Lustre: Skipped 1 previous similar message [ 7863.339139] Lustre: server umount lustre-MDT0000 complete [ 7865.825366] LustreError: 180376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7865.835755] LustreError: 180376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7867.370841] LustreError: 180359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448988 with bad export cookie 4731375782122615277 [ 7867.387982] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7867.398949] LustreError: 180359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7867.836971] Lustre: server umount lustre-MDT0001 complete [ 7882.625239] Lustre: server umount lustre-OST0000 complete [ 7891.925509] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 7902.052728] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7940.747052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7941.248381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7941.331827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7941.532542] Lustre: lustre-MDT0000: new disk, initializing [ 7941.743135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7941.795545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7947.197785] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7958.972655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7959.151383] Lustre: 185719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7959.219241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7959.223625] Lustre: Skipped 1 previous similar message [ 7959.321066] Lustre: lustre-MDT0001: new disk, initializing [ 7959.441043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7959.480469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7959.497699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7964.587628] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7976.423144] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7984.402466] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7992.494459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7992.808787] Lustre: lustre-OST0000: new disk, initializing [ 7992.813863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7992.821588] Lustre: 187686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7992.920726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8000.147428] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8001.101159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8001.106997] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8001.190328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8009.394523] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8013.891284] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8014.087810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8018.863961] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8019.031612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8021.985414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8021.987981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8021.991047] LustreError: Skipped 1 previous similar message [ 8022.005264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8022.005992] Lustre: Skipped 2 previous similar messages [ 8022.026753] Lustre: Skipped 3 previous similar messages [ 8025.069447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8025.080222] Lustre: Skipped 1 previous similar message [ 8030.178479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8030.192398] Lustre: Skipped 1 previous similar message [ 8035.296265] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8035.305439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8035.312354] Lustre: Skipped 1 previous similar message [ 8035.526099] Lustre: server umount lustre-OST0000 complete [ 8040.422571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8040.434843] Lustre: Skipped 1 previous similar message [ 8045.539700] LustreError: 186897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8045.678503] Lustre: server umount lustre-MDT0000 complete [ 8050.659835] LustreError: 185729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.675378] LustreError: 185729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8050.869582] LustreError: 185710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449171 with bad export cookie 4731375782122615998 [ 8050.886350] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8050.889473] LustreError: 185710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8051.206351] Lustre: server umount lustre-MDT0001 complete [ 8061.643499] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:26:20 (1788449180) [ 8078.532917] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 8081.509455] Key type lgssc unregistered [ 8081.782872] LNet: 190637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8081.795816] LNetError: 190637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8081.826774] LNet: Removed LNI 192.168.203.115@tcp [ 8082.628325] Key type .llcrypt unregistered [ 8082.631308] Key type ._llcrypt unregistered [ 8102.965565] Key type ._llcrypt registered [ 8102.970323] Key type .llcrypt registered [ 8103.107326] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8104.389246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8104.445425] alg: No test for adler32 (adler32-zlib) [ 8105.478721] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8105.787851] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 8107.496183] Key type lgssc registered [ 8108.735679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8116.477282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8116.495473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8117.976505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8122.685842] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8127.459668] LustreError: 191858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8127.482171] LustreError: 191858:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8131.929312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8132.389702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8137.027272] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8145.881286] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8152.668650] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8161.393452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8161.656436] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8161.661858] LustreError: 193724:0:(llog.c:1696:llog_backup()) MGC192.168.203.115@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8161.667829] Lustre: 193724:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.115@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8161.793279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8167.959893] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8175.496497] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8178.657216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8178.668710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.688269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8182.244739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8182.252392] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8182.261188] Lustre: Skipped 1 previous similar message [ 8182.280606] Lustre: Skipped 1 previous similar message [ 8183.710503] Lustre: server umount lustre-MDT0000 complete [ 8187.019427] LustreError: 191839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449307 with bad export cookie 2926371591038325625 [ 8187.021620] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8187.031079] LustreError: 191839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8187.394388] Lustre: server umount lustre-MDT0001 complete [ 8200.630263] Lustre: server umount lustre-OST0000 complete [ 8210.502214] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 8220.148340] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8253.399316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8253.606668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8253.636246] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8253.746139] Lustre: lustre-MDT0000: new disk, initializing [ 8253.830054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8253.844377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8258.564763] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8268.998886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.189336] Lustre: 197247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8269.310402] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8269.318566] Lustre: Skipped 1 previous similar message [ 8269.323596] LustreError: 198028:0:(llog.c:1696:llog_backup()) MGC192.168.203.115@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8269.337320] LustreError: 198028:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8269.338819] LustreError: 197257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8269.344774] Lustre: 198028:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.115@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8269.386684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8269.394116] Lustre: Skipped 1 previous similar message [ 8269.469426] Lustre: lustre-MDT0001: new disk, initializing [ 8269.596827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8269.642414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8269.652811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8274.910740] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8286.486797] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8293.508681] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8302.482389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8302.937306] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8302.941112] Lustre: Skipped 1 previous similar message [ 8302.948075] LustreError: 199191:0:(llog.c:1696:llog_backup()) MGC192.168.203.115@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8302.955638] LustreError: 199191:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8302.961773] Lustre: 199191:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.115@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8303.000928] Lustre: 199191:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8303.034689] Lustre: lustre-OST0000: new disk, initializing [ 8303.039910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8303.048240] Lustre: 199215:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8303.185635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8310.366519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8310.388759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8310.492631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8310.738912] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8320.614597] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8325.316577] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8325.699303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8331.257218] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8331.490475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8336.365805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8336.367973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8336.377185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8336.377192] Lustre: Skipped 1 previous similar message [ 8336.413331] Lustre: Skipped 1 previous similar message [ 8340.147625] Lustre: server umount lustre-OST0000 complete [ 8346.595519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8346.598698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8346.599371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8346.599386] LustreError: Skipped 1 previous similar message [ 8346.609197] Lustre: Skipped 1 previous similar message [ 8346.626588] Lustre: Skipped 2 previous similar messages [ 8351.106109] Lustre: server umount lustre-MDT0000 complete [ 8351.724330] LustreError: 197258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8351.769941] LustreError: 197258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8355.316350] LustreError: 197238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449476 with bad export cookie 2926371591038326339 [ 8355.339827] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8355.340368] LustreError: 197238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8355.770534] Lustre: server umount lustre-MDT0001 complete [ 8365.758135] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:31:24 (1788449484) [ 8383.869345] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8394.449705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8394.819437] LustreError: 202345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8394.888432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8399.505474] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8400.359078] LustreError: 202346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8407.601216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8412.181230] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8415.216964] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8422.111508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8428.783730] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8436.958603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8437.108079] Lustre: lustre-OST0001: new disk, initializing [ 8437.111516] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8437.118765] Lustre: 204654:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8437.220332] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8437.226170] Lustre: Skipped 2 previous similar messages [ 8438.704544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8438.709916] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8438.822117] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8443.840560] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8455.152475] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8462.853931] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8464.781536] Lustre: DEBUG MARKER: ADD OST3 [ 8469.610039] loop: module loaded [ 8469.660190] loop0: detected capacity change from 0 to 204800000 [ 8469.879957] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8469.896784] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8469.908494] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8476.558464] loop0: detected capacity change from 0 to 204800000 [ 8477.294650] Lustre: DEBUG MARKER: STOP OST3 [ 8483.240069] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8483.530029] Lustre: lustre-OST0003: new disk, initializing [ 8483.546857] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8483.565189] Lustre: 206679:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8485.209565] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8485.381549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8485.402426] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8485.451156] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8487.264693] LustreError: 206792:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8488.103519] Lustre: Failing over lustre-OST0003 [ 8488.433558] Lustre: server umount lustre-OST0003 complete [ 8490.458270] LustreError: 204747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8490.481971] LustreError: 204747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8490.499704] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8490.508793] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8492.093358] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8500.216626] Lustre: DEBUG MARKER: START OST3 again [ 8507.349067] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8507.659542] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8507.663814] Lustre: Skipped 1 previous similar message [ 8507.678935] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8508.993310] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8509.686841] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8509.711522] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8515.615716] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8538.716174] Lustre: Failing over lustre-OST0003 [ 8538.772339] Lustre: server umount lustre-OST0003 complete [ 8540.648066] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8540.648749] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8540.671106] LustreError: Skipped 1 previous similar message [ 8540.673975] LustreError: 204747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8540.673989] LustreError: 204747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 8540.746313] Lustre: Skipped 2 previous similar messages [ 8556.514826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8556.539822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8559.097669] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8559.127061] Lustre: Skipped 3 previous similar messages [ 8562.626177] Lustre: server umount lustre-MDT0000 complete [ 8567.386048] LustreError: 202327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449688 with bad export cookie 2926371591038327228 [ 8567.408767] LustreError: 202327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8567.411181] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8569.316846] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8569.327840] Lustre: Skipped 5 previous similar messages [ 8574.115739] Lustre: server umount lustre-MDT0001 complete [ 8579.102106] Lustre: server umount lustre-OST0000 complete [ 8584.103120] Lustre: server umount lustre-OST0001 complete [ 8591.534900] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 8599.837086] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8639.851461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8640.174805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8640.209355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8640.365719] Lustre: lustre-MDT0000: new disk, initializing [ 8640.483981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8640.519708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8644.869674] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8656.152299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8656.368805] Lustre: 212112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8656.428986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8656.434823] Lustre: Skipped 1 previous similar message [ 8656.524748] Lustre: lustre-MDT0001: new disk, initializing [ 8656.709750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8656.719843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8662.290932] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8672.370303] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8678.253597] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8685.068723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8685.391984] Lustre: 214083:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8686.815757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8686.952915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8691.601153] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8701.890355] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8707.778280] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8708.057895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8713.850286] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8714.150953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8717.795364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8717.796953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8717.804232] LustreError: Skipped 1 previous similar message [ 8717.831122] Lustre: Skipped 2 previous similar messages [ 8717.840152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8717.870630] Lustre: Skipped 3 previous similar messages [ 8722.924630] LustreError: 214066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8722.938268] LustreError: 214066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 8723.124627] Lustre: server umount lustre-OST0000 complete [ 8741.344140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8741.634976] Lustre: server umount lustre-MDT0000 complete [ 8745.077165] LustreError: 212106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449865 with bad export cookie 2926371591038329524 [ 8745.084179] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8745.093941] LustreError: 212106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8745.379561] Lustre: server umount lustre-MDT0001 complete [ 8749.237166] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:37:48 (1788449868) [ 8765.602679] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 8776.360863] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8816.326085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8816.662779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8816.708174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8816.716848] Lustre: Skipped 1 previous similar message [ 8816.842143] Lustre: lustre-MDT0000: new disk, initializing [ 8816.847973] Lustre: Skipped 1 previous similar message [ 8816.959872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8816.965342] Lustre: Skipped 1 previous similar message [ 8823.021696] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8835.497534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8835.631134] Lustre: 218701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8836.103596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8840.429125] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8850.228434] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8856.765943] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8864.933779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8865.352646] Lustre: 220668:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8867.310923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8872.801265] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8883.522407] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8889.192362] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8889.462122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8894.887716] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8895.152188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8898.029880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8898.030161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8898.044881] LustreError: Skipped 1 previous similar message [ 8898.052952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8898.063300] Lustre: Skipped 4 previous similar messages [ 8898.084451] Lustre: Skipped 7 previous similar messages [ 8903.436253] Lustre: server umount lustre-OST0000 complete [ 8914.253073] Lustre: server umount lustre-MDT0000 complete [ 8918.233920] LustreError: 218693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450038 with bad export cookie 2926371591038330406 [ 8918.236591] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8918.249209] LustreError: 218693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8918.512815] Lustre: server umount lustre-MDT0001 complete [ 8927.517229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8928.059293] LustreError: 222883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8928.084262] LustreError: 222883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8928.162486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8928.171583] Lustre: Skipped 5 previous similar messages [ 8932.340877] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8941.226976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8946.365209] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8955.056614] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8962.691420] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8971.534242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8978.445041] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8987.599862] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8998.369488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8998.381670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8998.385795] Lustre: Skipped 6 previous similar messages [ 9004.536248] Lustre: server umount lustre-OST0000 complete [ 9014.912714] Lustre: server umount lustre-MDT0000 complete [ 9018.510285] LustreError: 222863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450139 with bad export cookie 2926371591038331288 [ 9018.516577] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9018.527950] LustreError: 222863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9018.998803] Lustre: server umount lustre-MDT0001 complete [ 9127.421756] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9130.262764] Key type lgssc unregistered [ 9130.578996] LNet: 226550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9130.592092] LNetError: 226550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9130.608981] LNet: Removed LNI 192.168.203.115@tcp [ 9131.492281] Key type .llcrypt unregistered [ 9131.496257] Key type ._llcrypt unregistered [ 9157.637995] Key type ._llcrypt registered [ 9157.642245] Key type .llcrypt registered [ 9157.754241] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 9172.852835] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9173.876781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9173.961485] alg: No test for adler32 (adler32-zlib) [ 9175.056619] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9175.275595] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 9176.945236] Key type lgssc registered [ 9178.125677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9210.394154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9210.421653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9211.625831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9211.655389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9211.720033] Lustre: lustre-MDT0000: new disk, initializing [ 9211.798145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9211.830897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9216.078472] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9226.195893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9226.291842] Lustre: 229907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9226.356803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9226.361115] Lustre: Skipped 1 previous similar message [ 9226.417796] Lustre: lustre-MDT0001: new disk, initializing [ 9226.515667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9226.546181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9226.580764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9230.583614] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9238.879420] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9244.357692] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9251.142973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9251.450534] Lustre: lustre-OST0000: new disk, initializing [ 9251.462205] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9251.469390] Lustre: 231874:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9251.578127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9257.715745] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9258.607088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9258.621126] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9258.721696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9267.412210] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9271.598548] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9271.781418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9275.910451] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9276.125509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9279.468536] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9279.475885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9279.490165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9279.499168] Lustre: Skipped 1 previous similar message [ 9284.504099] Lustre: server umount lustre-OST0000 complete [ 9288.161263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9288.175457] LustreError: Skipped 1 previous similar message [ 9288.178983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9288.194841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9288.197401] Lustre: Skipped 1 previous similar message [ 9289.701662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9289.704266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9289.735480] Lustre: Skipped 1 previous similar message [ 9294.041087] Lustre: server umount lustre-MDT0000 complete [ 9297.706668] LustreError: 229899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450418 with bad export cookie 8673384863383660606 [ 9297.717313] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9297.720951] LustreError: 229899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9297.996425] Lustre: server umount lustre-MDT0001 complete [ 9316.743444] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9326.496645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9326.848188] LustreError: 234808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9326.878693] LustreError: 234808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9326.897606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9331.164783] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9332.195952] LustreError: 234809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9336.290092] LustreError: 234808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9339.761802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9340.009550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9344.251640] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9347.405410] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9353.422434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9353.711858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9359.114967] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9367.549047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9367.729599] Lustre: lustre-OST0001: new disk, initializing [ 9367.738165] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9367.739644] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 9367.743315] Lustre: 237118:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9371.974471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9371.988548] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9372.039276] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9373.431663] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9384.718356] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9393.044367] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9406.897750] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:48:45 (1788450525) [ 9411.559017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9411.570070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9411.588640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9423.328927] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9423.335114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9423.348567] Lustre: Skipped 11 previous similar messages [ 9423.540867] Lustre: server umount lustre-MDT0000 complete [ 9426.856830] LustreError: 237942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450547 with bad export cookie 8673384863383661488 [ 9426.871665] LustreError: 237942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9426.872519] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9427.144275] Lustre: server umount lustre-MDT0001 complete [ 9440.838393] Lustre: server umount lustre-OST0000 complete [ 9454.776510] Lustre: server umount lustre-OST0001 complete [ 9461.892249] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 9472.302753] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9511.645889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9511.827405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9511.880292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9511.965843] Lustre: lustre-MDT0000: new disk, initializing [ 9512.011904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9512.025640] Lustre: Skipped 1 previous similar message [ 9512.049536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9517.018861] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9529.380075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9529.504524] Lustre: 241067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9529.555139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9529.558383] Lustre: Skipped 1 previous similar message [ 9529.669367] Lustre: lustre-MDT0001: new disk, initializing [ 9529.849817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9529.868344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9535.034866] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9545.681708] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9551.874561] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9559.343550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9559.660858] Lustre: lustre-OST0000: new disk, initializing [ 9559.663829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9559.669224] Lustre: 243035:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9559.789715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9559.804640] Lustre: Skipped 1 previous similar message [ 9560.832322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9560.841640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9560.982930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9566.815163] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9576.873448] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9582.166324] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9582.506893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9587.599648] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9587.896293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9591.785680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9591.792545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9591.800689] Lustre: Skipped 4 previous similar messages [ 9591.801462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9591.830945] Lustre: Skipped 1 previous similar message [ 9596.262056] Lustre: server umount lustre-OST0000 complete [ 9601.246318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9607.566118] Lustre: server umount lustre-MDT0000 complete [ 9611.757557] LustreError: 243342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9611.800193] LustreError: 243342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9612.679407] LustreError: 241057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450733 with bad export cookie 8673384863383663616 [ 9612.685608] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9612.692292] LustreError: 241057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9613.057248] Lustre: server umount lustre-MDT0001 complete [ 9623.039362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9623.750425] LustreError: 245251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9623.899845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9628.754479] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9634.275324] LustreError: 245252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9634.311207] LustreError: 245252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9637.922043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9642.772072] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9649.866941] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9656.303849] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9663.019594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9669.626075] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9676.887671] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9822.189600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9822.195343] LustreError: Skipped 1 previous similar message [ 9822.199212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9822.209396] Lustre: Skipped 1 previous similar message [ 9822.214153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9822.217936] Lustre: Skipped 4 previous similar messages [ 9827.656640] Lustre: server umount lustre-OST0000 complete [ 9837.560588] Lustre: server umount lustre-MDT0000 complete [ 9841.143626] LustreError: 245231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450961 with bad export cookie 8673384863383664505 [ 9841.145701] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9841.157468] LustreError: 245231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9841.456221] Lustre: server umount lustre-MDT0001 complete [ 9945.940380] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9948.471956] Key type lgssc unregistered [ 9948.808669] LNet: 248866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9948.821846] LNetError: 248866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9948.847156] LNet: Removed LNI 192.168.203.115@tcp [ 9949.864176] Key type .llcrypt unregistered [ 9949.866195] Key type ._llcrypt unregistered [ 9975.112371] Key type ._llcrypt registered [ 9975.116543] Key type .llcrypt registered [ 9975.213083] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 9989.168870] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9990.137588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9990.162407] alg: No test for adler32 (adler32-zlib) [ 9991.228369] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9991.459713] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 9993.136403] Key type lgssc registered [ 9994.056153] Lustre: Echo OBD driver; http://www.lustre.org/ [10035.265334] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10045.258800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10045.300522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10046.567907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10046.608422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10046.718979] Lustre: lustre-MDT0000: new disk, initializing [10046.828121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10046.854210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10051.368533] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10061.812361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10061.915209] Lustre: 252935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10061.960568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10061.964811] Lustre: Skipped 1 previous similar message [10062.065940] Lustre: lustre-MDT0001: new disk, initializing [10062.177561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10062.205755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10062.218833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10066.816504] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10071.751712] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10078.271420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10078.526634] Lustre: lustre-OST0000: new disk, initializing [10078.530896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10078.535062] Lustre: 254566:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10078.607719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10079.348273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10079.357389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10079.539899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10085.929202] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10096.094420] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10096.282631] Lustre: lustre-OST0001: new disk, initializing [10096.285920] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10096.296286] Lustre: 255437:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10096.414768] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10102.940386] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10105.458896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10105.477082] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10105.547683] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10114.427961] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10121.661161] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10133.985954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10134.005274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10134.027044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10136.545530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10136.551449] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10136.567970] Lustre: Skipped 2 previous similar messages [10138.624855] Lustre: server umount lustre-MDT0000 complete [10142.167973] LustreError: 252928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451262 with bad export cookie 5689123032752814476 [10142.172297] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10142.176079] LustreError: 252928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10142.696252] Lustre: server umount lustre-MDT0001 complete [10156.137680] Lustre: server umount lustre-OST0000 complete [10170.043799] Lustre: server umount lustre-OST0001 complete [10183.038214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10192.841446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10200.857221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10210.866104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10231.050392] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10240.248992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10240.269856] Lustre: lustre-MDT0000: reset Object Index mappings [10240.662617] LustreError: 260462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10240.678718] LustreError: 260462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10240.739920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10245.195668] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10246.118311] LustreError: 260463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10251.242023] LustreError: 260462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10253.857519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10253.878413] Lustre: lustre-MDT0001: reset Object Index mappings [10254.381208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10259.165690] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10262.262571] Lustre: 261604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10268.780622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10275.942339] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10278.376989] LustreError: 261958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10283.514361] LustreError: 261958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10285.382612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10285.565900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10285.708916] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10285.719597] Lustre: Skipped 1 previous similar message [10291.213279] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10292.600362] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10301.331732] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10305.033836] Lustre: 263475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10310.625957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10310.632804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10310.649531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10310.656077] Lustre: Skipped 3 previous similar messages [10312.596722] Lustre: server umount lustre-MDT0000 complete [10316.780518] LustreError: 260463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10316.793322] LustreError: 260442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451437 with bad export cookie 5689123032752821266 [10316.801225] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10316.816903] LustreError: 260463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10316.879439] LustreError: 260442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10317.396819] Lustre: server umount lustre-MDT0001 complete [10332.246808] Lustre: server umount lustre-OST0000 complete [10346.874816] Lustre: server umount lustre-OST0001 complete [10368.857839] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:04:47 (1788451487) [10386.161481] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [10396.474231] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10434.147262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10434.390482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10434.434201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10434.513398] Lustre: lustre-MDT0000: new disk, initializing [10434.589654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10434.609514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10439.244324] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10450.117934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10450.240857] Lustre: 267731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10450.291397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10450.297623] Lustre: Skipped 1 previous similar message [10450.382339] Lustre: lustre-MDT0001: new disk, initializing [10450.511953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10450.537241] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10454.903714] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10464.298461] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10470.480405] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10477.383262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10477.617221] Lustre: lustre-OST0000: new disk, initializing [10477.620577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10477.625732] Lustre: 269701:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10479.190358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10479.205124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10479.400838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10484.468307] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10494.168415] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10499.234928] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10499.555233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10504.579259] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10504.812057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10510.306651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10510.316587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10510.318907] LustreError: Skipped 1 previous similar message [10510.321793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10510.321800] Lustre: Skipped 4 previous similar messages [10510.362408] Lustre: Skipped 4 previous similar messages [10513.157644] Lustre: server umount lustre-OST0000 complete [10520.548111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10520.553473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10520.568938] Lustre: Skipped 1 previous similar message [10520.578973] Lustre: Skipped 1 previous similar message [10523.412580] Lustre: server umount lustre-MDT0000 complete [10525.665816] LustreError: 269289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10525.690727] LustreError: 269289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10527.173264] LustreError: 267723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451647 with bad export cookie 5689123032752823359 [10527.177477] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10527.182442] LustreError: 267723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10527.596107] Lustre: server umount lustre-MDT0001 complete [10544.719599] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10554.803540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10555.233553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10555.238438] Lustre: Skipped 2 previous similar messages [10559.213650] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10560.485734] LustreError: 272635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10560.508340] LustreError: 272635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10568.733514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10573.534883] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10577.059507] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10584.576132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10591.192673] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10600.824494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10601.093532] Lustre: lustre-OST0001: new disk, initializing [10601.100808] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10601.113263] Lustre: 274943:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10602.544783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10602.557539] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10602.684623] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10607.527519] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10618.311024] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10622.499617] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10628.659570] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10628.703864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10638.819481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10638.828400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10638.840713] Lustre: Skipped 3 previous similar messages [10638.851574] Lustre: Skipped 4 previous similar messages [10642.513573] Lustre: server umount lustre-MDT0000 complete [10643.943994] LustreError: 272635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10643.960689] LustreError: 272635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10646.343097] LustreError: 275705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451767 with bad export cookie 5689123032752824234 [10646.346341] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10646.359213] LustreError: 275705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10646.762970] Lustre: server umount lustre-MDT0001 complete [10661.147096] Lustre: server umount lustre-OST0000 complete [10664.416135] Lustre: 250445:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451769/real 1788451769] req@ffff9fc238302300 x1875326913452416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788451785 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10664.453308] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10665.044773] Lustre: server umount lustre-OST0001 complete [10684.278157] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10694.792562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10695.203343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10695.207573] Lustre: Skipped 3 previous similar messages [10699.640899] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10708.381845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10713.604176] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10716.983377] Lustre: 279138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10724.409794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10732.153458] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10741.241966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10741.366497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10746.904570] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10747.990459] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10756.120494] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10760.046556] Lustre: 281015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10765.368381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10765.409182] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10775.713528] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:11:34 (1788451894) [10792.935083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10792.944646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10792.962960] Lustre: Skipped 2 previous similar messages [10792.971876] Lustre: Skipped 2 previous similar messages [10793.653195] Lustre: server umount lustre-MDT0000 complete [10798.058806] LustreError: 279009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10798.087504] LustreError: 279009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10798.112627] LustreError: 279140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451918 with bad export cookie 5689123032752830779 [10798.128607] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10798.144922] LustreError: 279140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10804.719927] Lustre: server umount lustre-MDT0001 complete [10814.753854] Lustre: server umount lustre-OST0000 complete [10819.592067] Lustre: server umount lustre-OST0001 complete [10827.973448] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [10838.302508] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10884.979819] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10896.377690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10896.691448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10896.774424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10896.924883] Lustre: lustre-MDT0000: new disk, initializing [10897.067886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10902.299407] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10913.431362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10913.568600] Lustre: 285254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10914.040758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10918.996931] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10930.777883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10931.233093] Lustre: 286888:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10933.004226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10937.729874] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10949.312727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10949.568230] Lustre: 287761:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10951.681859] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10951.691815] Lustre: Skipped 1 previous similar message [10951.802334] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10957.417548] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10967.603200] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10971.428091] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10971.434074] Lustre: Skipped 1 previous similar message [10987.032218] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:15:05 (1788452105) [10991.073379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10991.088889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10991.109840] Lustre: Skipped 3 previous similar messages [10991.118832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10991.123464] Lustre: Skipped 2 previous similar messages [10991.449596] Lustre: server umount lustre-MDT0000 complete [10995.424022] LustreError: 285249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452116 with bad export cookie 5689123032752838430 [10995.428265] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10995.435353] LustreError: 285249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10995.846800] Lustre: server umount lustre-MDT0001 complete [11009.974730] Lustre: server umount lustre-OST0000 complete [11024.398807] Lustre: server umount lustre-OST0001 complete [11033.112937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11033.702387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11033.706140] Lustre: Skipped 7 previous similar messages [11038.673133] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11049.042531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11055.773415] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11063.496916] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11069.854775] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11077.804829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11079.235752] LustreError: 291837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11079.275589] LustreError: 291837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11085.117548] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11092.210321] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11111.516739] loop1: detected capacity change from 0 to 204800000 [11111.759671] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11111.787801] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11111.814449] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11116.836419] loop1: detected capacity change from 0 to 204800000 [11122.969394] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11123.083416] LustreError: 289993:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs10516: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11123.109202] LustreError: 289993:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs10516 [11123.130256] LustreError: 289993:0:(mgs_handler.c:517:mgs_target_reg()) fs10516-MDT0001: Error checking target index: rc = -13 [11123.158850] LustreError: fs10516-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11123.168698] LustreError: 293727:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11123.184164] LustreError: 293727:0:(tgt_mount.c:2129:server_put_super()) no obd fs10516-MDT0001 [11123.194442] LustreError: 293727:0:(tgt_mount.c:116:server_deregister_mount()) fs10516-MDT0001 not registered [11123.245981] Lustre: server umount fs10516-MDT0001 complete [11123.251372] LustreError: 293727:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11131.966764] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11132.088352] Lustre: Setting parameter fs10516-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs10516-MDT0001 [11132.161996] Lustre: srv-fs10516-MDT0001: No data found on store. Initialize space. [11132.174529] Lustre: Skipped 4 previous similar messages [11132.273365] Lustre: fs10516-MDT0001: new disk, initializing [11132.280297] Lustre: Skipped 3 previous similar messages [11132.411517] Lustre: srv-fs10516-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11132.417389] Lustre: 294193:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10516-MDT0001-mdtlov: can't create new llog: rc = -115 [11132.429653] LustreError: 294193:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10516-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11137.008798] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11143.337227] Lustre: server umount fs10516-MDT0001 complete [11155.583212] loop1: detected capacity change from 0 to 204800000 [11160.756561] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11161.063492] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11161.142053] Lustre: 295758:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10516-MDT0001-mdtlov: can't create new llog: rc = -115 [11161.156184] LustreError: 295758:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10516-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11165.284745] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11170.367742] Lustre: server umount fs10516-MDT0001 complete [11176.430919] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11183.073742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11183.088352] Lustre: Skipped 3 previous similar messages [11183.093443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11188.601134] Lustre: server umount lustre-MDT0000 complete [11199.011122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11199.303430] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11205.117611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11206.161807] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11214.947471] loop1: detected capacity change from 0 to 204800000 [11220.931472] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11221.499323] Lustre: 298206:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10516-MDT0001-mdtlov: can't create new llog: rc = -115 [11221.512230] LustreError: 298206:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10516-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11226.266160] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11231.799867] Lustre: server umount fs10516-MDT0001 complete [11256.297506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11256.305452] LustreError: Skipped 1 previous similar message [11261.249166] Lustre: server umount lustre-OST0000 complete [11270.916308] Lustre: server umount lustre-MDT0000 complete [11274.834553] LustreError: 289985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452395 with bad export cookie 5689123032752841111 [11274.852125] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11275.295604] Lustre: server umount lustre-MDT0001 complete [11286.132388] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [11289.133808] Key type lgssc unregistered [11289.478427] LNet: 300555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11289.490288] LNetError: 300555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11289.508652] LNet: Removed LNI 192.168.203.115@tcp [11290.503171] Key type .llcrypt unregistered [11290.505214] Key type ._llcrypt unregistered [11310.984801] Key type ._llcrypt registered [11310.989342] Key type .llcrypt registered [11311.166780] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [11312.102252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11312.118890] alg: No test for adler32 (adler32-zlib) [11313.500382] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11313.957282] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [11315.761846] Key type lgssc registered [11316.910857] Lustre: Echo OBD driver; http://www.lustre.org/ [11326.944293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11326.963730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11328.550195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11333.610545] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11338.723432] LustreError: 301827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.749674] LustreError: 301827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11343.383426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11343.873576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11348.636961] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11357.546110] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11364.405548] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11371.946578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11372.379981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11373.426994] LustreError: 303657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11378.670393] LustreError: 303660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11379.229421] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11383.792739] LustreError: 303657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11387.849579] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11388.918360] LustreError: 303660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11388.939487] LustreError: 303660:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11399.156860] LustreError: 303657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11399.176788] LustreError: 303657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11406.022296] loop1: detected capacity change from 0 to 204800000 [11406.187696] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11406.194740] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11406.205948] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11411.531042] loop1: detected capacity change from 0 to 204800000 [11416.515708] LustreError: 303657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11416.550815] LustreError: 303657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [11418.158417] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11418.324471] Lustre: fs452-OST0000: new disk, initializing [11418.336844] Lustre: srv-fs452-OST0000: No data found on store. Initialize space. [11418.352922] Lustre: 305566:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11418.450200] Lustre: fs452-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11424.846722] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11434.721634] Lustre: server umount fs452-OST0000 complete [11449.206578] loop1: detected capacity change from 0 to 204800000 [11452.320221] LustreError: 303657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11452.340986] LustreError: 303657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [11453.418160] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11455.030132] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11455.260044] Lustre: fs452-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11461.805798] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11468.794883] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11470.927436] loop2: detected capacity change from 0 to 204800000 [11471.153599] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11471.174371] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11471.195620] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11477.041063] loop2: detected capacity change from 0 to 204800000 [11482.730817] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11482.847555] LustreError: fs452-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11482.855787] LustreError: fs452-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11482.862316] LustreError: 308610:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11482.867528] LustreError: 308610:0:(tgt_mount.c:2129:server_put_super()) no obd fs452-OST0001 [11482.872454] LustreError: 308610:0:(tgt_mount.c:116:server_deregister_mount()) fs452-OST0001 not registered [11482.904643] Lustre: server umount fs452-OST0001 complete [11482.907234] LustreError: 308610:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11493.797558] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11494.014408] Lustre: fs452-OST0001: new disk, initializing [11494.021541] Lustre: srv-fs452-OST0001: No data found on store. Initialize space. [11494.029457] Lustre: 309160:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11494.149198] Lustre: fs452-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11500.518530] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11508.625435] Lustre: server umount fs452-OST0000 complete [11517.214306] Lustre: server umount fs452-OST0001 complete [11518.887659] LustreError: 303660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11518.902521] LustreError: 303660:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11529.443569] loop1: detected capacity change from 0 to 204800000 [11534.921740] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11535.262869] Lustre: fs452-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11541.704876] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11548.910661] loop2: detected capacity change from 0 to 204800000 [11555.115678] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11561.734330] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11571.391637] Lustre: server umount fs452-OST0000 complete [11580.229313] Lustre: server umount fs452-OST0001 complete [11593.698594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11593.702725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11593.709425] LustreError: Skipped 1 previous similar message [11593.731980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11593.738557] Lustre: Skipped 1 previous similar message [11596.771048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11598.165591] Lustre: server umount lustre-OST0000 complete [11604.963501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11604.975290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11604.983368] Lustre: Skipped 1 previous similar message [11604.990163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11604.994563] Lustre: Skipped 1 previous similar message [11607.010135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11607.010450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11608.421660] Lustre: server umount lustre-MDT0000 complete [11612.007385] LustreError: 301808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452732 with bad export cookie 5475638758117943121 [11612.026260] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11612.029286] LustreError: 301808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11612.377967] Lustre: server umount lustre-MDT0001 complete [11623.366551] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [11626.129422] Key type lgssc unregistered [11626.444726] LNet: 315089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11626.450403] LNetError: 315089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11627.496620] LNet: Removed LNI 192.168.203.115@tcp [11628.473141] Key type .llcrypt unregistered [11628.475423] Key type ._llcrypt unregistered [11663.043157] Key type ._llcrypt registered [11663.054511] Key type .llcrypt registered [11663.244805] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [11681.092547] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [11682.170103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11682.194926] alg: No test for adler32 (adler32-zlib) [11683.254111] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11683.404624] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [11685.040175] Key type lgssc registered [11686.312582] Lustre: Echo OBD driver; http://www.lustre.org/ [11724.048897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11724.079557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11725.339861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11725.381292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11725.463326] Lustre: lustre-MDT0000: new disk, initializing [11725.550245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11725.563955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11730.482684] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11741.608914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11741.731994] Lustre: 318951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11741.781386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11741.784798] Lustre: Skipped 1 previous similar message [11741.850396] Lustre: lustre-MDT0001: new disk, initializing [11741.944298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11741.981320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11741.993296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11747.222276] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11757.394893] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11763.107727] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11769.646324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11770.009743] Lustre: lustre-OST0000: new disk, initializing [11770.016178] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11770.026181] Lustre: 320921:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11770.161620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11775.975326] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11776.631888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11776.662881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11776.885674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11786.957451] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11789.361316] Lustre: Mounted lustre-client - version 2.17.56_51_g2aa7cea [11790.595457] Lustre: Unmounted lustre-client [11804.659813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11804.665673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11804.681480] Lustre: Skipped 1 previous similar message [11809.837651] Lustre: server umount lustre-OST0000 complete [11814.882562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11814.890280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11814.900904] Lustre: Skipped 1 previous similar message [11814.913058] Lustre: Skipped 1 previous similar message [11820.005444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11820.027434] Lustre: Skipped 2 previous similar messages [11820.389138] Lustre: server umount lustre-MDT0000 complete [11824.339130] LustreError: 318943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452945 with bad export cookie 8090868025854404866 [11824.353914] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11825.303816] Lustre: server umount lustre-MDT0001 complete [11839.136188] LNet: 1 local NIs in recovery (showing 1): 192.168.203.115@tcp [11842.181544] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [11845.383186] Key type lgssc unregistered [11845.681512] LNet: 323085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11845.691059] LNetError: 323085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11920.487461] LNet: Removed LNI 192.168.203.115@tcp [11921.542792] Key type .llcrypt unregistered [11921.544824] Key type ._llcrypt unregistered [11941.433354] Key type ._llcrypt registered [11941.435818] Key type .llcrypt registered [11941.577289] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [11961.541816] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [11963.665961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11963.719862] alg: No test for adler32 (adler32-zlib) [11965.030400] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11965.424911] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [11967.128412] Key type lgssc registered [11968.655552] Lustre: Echo OBD driver; http://www.lustre.org/ [12007.906666] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:32:06 (1788453126) [12015.589664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12015.631139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12016.928486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12016.982564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12017.075816] Lustre: lustre-MDT0000: new disk, initializing [12017.162224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12017.191283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12022.748496] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12033.942787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12034.092709] Lustre: 326308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12034.157850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12034.164342] Lustre: Skipped 1 previous similar message [12034.255291] Lustre: lustre-MDT0001: new disk, initializing [12034.384553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12034.445421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12034.470055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12039.365490] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12050.276668] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12056.065802] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12063.357577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12063.647677] Lustre: lustre-OST0000: new disk, initializing [12063.651544] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12063.658948] Lustre: 328280:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12063.747562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12070.928797] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12073.092298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12073.104639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12073.292454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12082.946220] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12104.164835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12104.166907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12104.179623] LustreError: Skipped 1 previous similar message [12104.181729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12104.194169] Lustre: Skipped 1 previous similar message [12104.217208] Lustre: Skipped 1 previous similar message [12105.633540] Lustre: server umount lustre-OST0000 complete [12111.334445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12111.348898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12111.377688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12114.402106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12114.403776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12114.441283] Lustre: Skipped 1 previous similar message [12115.740941] Lustre: server umount lustre-MDT0000 complete [12119.772463] LustreError: 326300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453240 with bad export cookie 14685900748770660697 [12119.780529] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12119.782832] LustreError: 326300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12120.242874] Lustre: server umount lustre-MDT0001 complete [12132.293487] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12135.409245] Key type lgssc unregistered [12135.724546] LNet: 330191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12135.735538] LNetError: 330191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12135.754317] LNet: Removed LNI 192.168.203.115@tcp [12136.680842] Key type .llcrypt unregistered [12136.684086] Key type ._llcrypt unregistered [12158.295242] Key type ._llcrypt registered [12158.297323] Key type .llcrypt registered [12158.400128] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12159.804671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12159.886572] alg: No test for adler32 (adler32-zlib) [12161.006867] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12161.267345] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [12162.928223] Key type lgssc registered [12164.161960] Lustre: Echo OBD driver; http://www.lustre.org/ [12172.398919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12172.415489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12173.913489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12178.990852] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12184.046405] LustreError: 331467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12184.082586] LustreError: 331467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12187.951215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12188.421202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12193.054221] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12200.916592] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12207.853357] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12215.752691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12216.357251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12225.486054] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12236.196649] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12247.400491] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12248.906860] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12254.180695] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12254.181535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12254.200165] Lustre: Skipped 1 previous similar message [12254.221368] Lustre: Skipped 2 previous similar messages [12259.321496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12259.333994] Lustre: Skipped 2 previous similar messages [12259.691965] Lustre: server umount lustre-MDT0000 complete [12259.767841] LustreError: 331461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12264.299855] LustreError: 331448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453385 with bad export cookie 4865345127484834867 [12264.309288] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12264.325355] LustreError: 331448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12264.417696] LustreError: 332480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12264.418527] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12264.467213] Lustre: Skipped 1 previous similar message [12264.470417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12264.474033] Lustre: Skipped 1 previous similar message [12269.544459] LustreError: 331461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12269.544659] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12269.561115] LustreError: 331461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12269.606961] Lustre: Skipped 1 previous similar message [12270.928542] Lustre: server umount lustre-MDT0001 complete [12276.468623] Lustre: server umount lustre-OST0000 complete [12288.606350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12294.037028] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12297.108652] Lustre: 335650:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.115@tcp [12299.506400] Lustre: server umount MGS complete [12308.773906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12309.470135] LustreError: 336223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12309.499456] LustreError: 336223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12309.643443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12315.596349] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12320.163216] LustreError: 336219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12320.197407] LustreError: 336219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12326.508820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12326.997817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12332.384831] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12341.491475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12341.991516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12349.275757] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12358.427186] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12360.046767] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12366.817802] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12369.163758] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12375.019360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12375.025589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12375.039561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12375.042226] Lustre: Skipped 1 previous similar message [12376.921493] Lustre: server umount lustre-OST0000 complete [12381.174514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12381.204796] Lustre: Skipped 1 previous similar message [12386.276972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12386.281335] Lustre: Skipped 5 previous similar messages [12387.267641] Lustre: server umount lustre-MDT0000 complete [12391.396714] LustreError: 336953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12391.430818] LustreError: 336953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12392.121894] LustreError: 336203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453512 with bad export cookie 4865345127484835707 [12392.134445] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12392.142512] LustreError: 336203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12392.570413] Lustre: server umount lustre-MDT0001 complete [12406.777876] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12410.334752] Key type lgssc unregistered [12410.772414] LNet: 339665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12410.786293] LNetError: 339665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12410.811799] LNet: Removed LNI 192.168.203.115@tcp [12412.082100] Key type .llcrypt unregistered [12412.085768] Key type ._llcrypt unregistered [12432.274460] Key type ._llcrypt registered [12432.277197] Key type .llcrypt registered [12432.385629] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [12448.607317] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12449.606065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12449.648748] alg: No test for adler32 (adler32-zlib) [12450.892713] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12451.146829] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [12452.800219] Key type lgssc registered [12454.015392] Lustre: Echo OBD driver; http://www.lustre.org/ [12492.713778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12492.736740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12493.986597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12494.011519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12494.068541] Lustre: lustre-MDT0000: new disk, initializing [12494.137290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12494.155443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12498.481636] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12509.493281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12509.614256] Lustre: 342827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12509.714388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12509.724950] Lustre: Skipped 1 previous similar message [12509.805534] Lustre: lustre-MDT0001: new disk, initializing [12509.909843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12509.959210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12509.984651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12514.303986] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12525.584563] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12532.447216] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12541.745770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12542.171636] Lustre: lustre-OST0000: new disk, initializing [12542.176364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12542.181603] Lustre: 344796:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12542.293983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12548.735226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12548.753781] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12548.944612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12549.595665] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12560.493680] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12566.129542] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12566.398417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12571.320938] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12571.573288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12574.689491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12574.691431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12574.701974] LustreError: Skipped 1 previous similar message [12574.706235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12574.721973] Lustre: Skipped 1 previous similar message [12574.750837] Lustre: Skipped 1 previous similar message [12580.015421] Lustre: server umount lustre-OST0000 complete [12584.931604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12584.934349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12584.949952] Lustre: Skipped 1 previous similar message [12584.965153] Lustre: Skipped 1 previous similar message [12590.050660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12590.062337] Lustre: Skipped 1 previous similar message [12595.173028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12595.187964] Lustre: Skipped 1 previous similar message [12598.240155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12598.580504] Lustre: server umount lustre-MDT0000 complete [12600.291520] LustreError: 344028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12600.320338] LustreError: 344028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12602.592558] LustreError: 342820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453723 with bad export cookie 17061089314400659277 [12602.602553] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12602.608231] LustreError: 342820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12602.965178] Lustre: server umount lustre-MDT0001 complete [12613.778517] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:42:12 (1788453732) [12632.485572] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [12643.527574] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12683.385520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12683.593497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12683.639600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12683.740404] Lustre: lustre-MDT0000: new disk, initializing [12683.870671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12683.899764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12688.695480] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12699.988479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12700.188457] Lustre: 349558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12700.256928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12700.261628] Lustre: Skipped 1 previous similar message [12700.359743] Lustre: lustre-MDT0001: new disk, initializing [12700.486546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12700.550765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12700.565733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12706.720018] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12718.123472] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12726.023618] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12735.392839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12735.625220] Lustre: lustre-OST0000: new disk, initializing [12735.634282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12735.647376] Lustre: 351528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12735.755596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12741.784366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12741.808473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12742.086166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12746.594901] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12759.657656] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12793.322684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12793.327638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12793.343878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12796.490098] Lustre: server umount lustre-OST0000 complete [12803.042111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12803.054151] LustreError: Skipped 1 previous similar message [12803.060580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12803.076741] Lustre: Skipped 1 previous similar message [12803.083187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12803.086238] Lustre: Skipped 1 previous similar message [12808.012825] Lustre: server umount lustre-MDT0000 complete [12808.677146] LustreError: 349565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12808.712464] LustreError: 349565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12811.823350] LustreError: 349548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453932 with bad export cookie 17061089314400660166 [12811.833917] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12811.846905] LustreError: 349548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12812.251847] Lustre: server umount lustre-MDT0001 complete [12822.396178] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12825.010675] Key type lgssc unregistered [12825.249757] LNet: 353683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12825.258965] LNetError: 353683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12825.294737] LNet: Removed LNI 192.168.203.115@tcp [12826.074622] Key type .llcrypt unregistered [12826.079726] Key type ._llcrypt unregistered [12847.301717] Key type ._llcrypt registered [12847.306914] Key type .llcrypt registered [12847.473030] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12848.811709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12848.873238] alg: No test for adler32 (adler32-zlib) [12850.096318] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12850.358684] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [12852.097099] Key type lgssc registered [12853.320153] Lustre: Echo OBD driver; http://www.lustre.org/ [12861.274489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12861.319904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12862.874989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12867.521449] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12873.189088] LustreError: 354970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12873.216222] LustreError: 354970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12876.030295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12876.355971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12880.810081] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12888.048331] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12893.452755] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12901.007418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12901.519697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12907.222452] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12914.360278] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12933.604962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12933.617894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12933.651849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12936.673184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12936.689544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12936.713360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12939.292468] Lustre: server umount lustre-OST0000 complete [12946.921132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12946.929155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12946.947461] Lustre: Skipped 1 previous similar message [12946.970132] Lustre: Skipped 2 previous similar messages [12949.717431] Lustre: server umount lustre-MDT0000 complete [12952.035170] LustreError: 354967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12952.065731] LustreError: 354967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12954.181407] LustreError: 354953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454074 with bad export cookie 4399006170671882539 [12954.195570] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12954.206275] LustreError: 354953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12955.043502] Lustre: server umount lustre-MDT0001 complete [12967.780983] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12971.955476] Key type lgssc unregistered [12972.286137] LNet: 360375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12972.299674] LNetError: 360375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12973.371770] LNet: Removed LNI 192.168.203.115@tcp [12974.731195] Key type .llcrypt unregistered [12974.738926] Key type ._llcrypt unregistered [12996.861748] Key type ._llcrypt registered [12996.872774] Key type .llcrypt registered [12997.020114] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12998.395903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12998.413088] alg: No test for adler32 (adler32-zlib) [12999.659941] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13000.104259] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [13001.881723] Key type lgssc registered [13003.435418] Lustre: Echo OBD driver; http://www.lustre.org/ [13013.576669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13013.620922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13015.413809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13022.219515] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [13025.775186] LustreError: 364141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13025.796848] LustreError: 364141:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13030.881974] LustreError: 364138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13033.013531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13033.647600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13039.539662] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [13050.089789] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13058.189521] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13066.321451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13066.798800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13072.727814] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [13080.690415] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13094.374082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13094.382278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13094.396762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13096.425318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13096.429076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13096.434446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13100.418905] Lustre: server umount lustre-OST0000 complete [13105.125784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13105.145716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13105.168477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13105.180841] Lustre: Skipped 1 previous similar message [13111.031708] Lustre: server umount lustre-MDT0000 complete [13111.778640] LustreError: 364138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13111.799853] LustreError: 364138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13114.855877] LustreError: 364070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454235 with bad export cookie 9572066287042741078 [13114.863917] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13114.871739] LustreError: 364070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13115.726831] Lustre: server umount lustre-MDT0001 complete [13128.577315] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13131.811178] Key type lgssc unregistered [13132.195873] LNet: 371946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13132.200957] LNetError: 371946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13132.217582] LNet: Removed LNI 192.168.203.115@tcp [13133.309821] Key type .llcrypt unregistered [13133.312166] Key type ._llcrypt unregistered [13152.947183] Key type ._llcrypt registered [13152.950200] Key type .llcrypt registered [13153.050577] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [13154.093597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13154.118920] alg: No test for adler32 (adler32-zlib) [13155.368384] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13155.628952] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [13157.329133] Key type lgssc registered [13158.937818] Lustre: Echo OBD driver; http://www.lustre.org/ [13169.288880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13169.309863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13171.196744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13176.607177] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [13180.396660] LustreError: 375706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13180.432530] LustreError: 375706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13185.506899] LustreError: 375707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13187.743595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13188.405660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13195.536960] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [13205.092864] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13210.869685] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13218.138782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13218.572420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13224.724262] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [13232.862393] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13256.673126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13256.686312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13256.688588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13256.706928] Lustre: Skipped 1 previous similar message [13260.886182] Lustre: server umount lustre-OST0000 complete [13265.382216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13265.395626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13265.418287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13265.425213] Lustre: Skipped 1 previous similar message [13266.915041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13266.915967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13266.948840] Lustre: Skipped 1 previous similar message [13270.778821] Lustre: server umount lustre-MDT0000 complete [13272.038439] LustreError: 375698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13272.060722] LustreError: 375698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13274.633839] LustreError: 375632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454395 with bad export cookie 12227793083491718410 [13274.637196] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13274.645682] LustreError: 375632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13274.926107] Lustre: server umount lustre-MDT0001 complete [13288.396715] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13292.045224] Key type lgssc unregistered [13292.439769] LNet: 381773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13292.457944] LNetError: 381773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13292.473974] LNet: Removed LNI 192.168.203.115@tcp [13293.845946] Key type .llcrypt unregistered [13293.849240] Key type ._llcrypt unregistered [13398.427240] Key type ._llcrypt registered [13398.429416] Key type .llcrypt registered [13398.549857] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13398.879092] Key type .llcrypt unregistered [13398.890888] Key type ._llcrypt unregistered [13479.036157] Key type ._llcrypt registered [13479.045682] Key type .llcrypt registered [13479.139759] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13479.466376] Key type .llcrypt unregistered [13479.469683] Key type ._llcrypt unregistered [13516.385229] Key type ._llcrypt registered [13516.389182] Key type .llcrypt registered [13516.508816] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13516.765411] Key type .llcrypt unregistered [13516.768221] Key type ._llcrypt unregistered [13549.049417] Key type ._llcrypt registered [13549.051431] Key type .llcrypt registered [13549.149179] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13549.483321] Key type .llcrypt unregistered [13549.487175] Key type ._llcrypt unregistered [13642.728678] Key type ._llcrypt registered [13642.730914] Key type .llcrypt registered [13642.898557] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13643.267429] Key type .llcrypt unregistered [13643.276171] Key type ._llcrypt unregistered [13675.404595] Key type ._llcrypt registered [13675.407217] Key type .llcrypt registered [13675.504281] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13675.961324] Key type .llcrypt unregistered [13675.964502] Key type ._llcrypt unregistered [13709.492414] Key type ._llcrypt registered [13709.494478] Key type .llcrypt registered [13709.662151] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13710.028788] Key type .llcrypt unregistered [13710.036128] Key type ._llcrypt unregistered [13750.422983] Key type ._llcrypt registered [13750.424914] Key type .llcrypt registered [13750.523439] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13750.980216] Key type .llcrypt unregistered [13750.983254] Key type ._llcrypt unregistered [13788.937175] Key type ._llcrypt registered [13788.938902] Key type .llcrypt registered [13789.155612] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13789.652364] Key type .llcrypt unregistered [13789.661248] Key type ._llcrypt unregistered [13825.586715] Key type ._llcrypt registered [13825.589558] Key type .llcrypt registered [13825.769562] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13826.205305] Key type .llcrypt unregistered [13826.207232] Key type ._llcrypt unregistered [13899.556596] Key type ._llcrypt registered [13899.559678] Key type .llcrypt registered [13899.676632] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13900.072501] Key type .llcrypt unregistered [13900.074685] Key type ._llcrypt unregistered [13941.949973] Key type ._llcrypt registered [13941.957762] Key type .llcrypt registered [13942.145921] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13942.670752] Key type .llcrypt unregistered [13942.677352] Key type ._llcrypt unregistered [13977.545515] Key type ._llcrypt registered [13977.548248] Key type .llcrypt registered [13977.682659] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13977.907790] Key type .llcrypt unregistered [13977.910598] Key type ._llcrypt unregistered [14046.862458] Key type ._llcrypt registered [14046.864654] Key type .llcrypt registered [14047.029317] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [14066.405413] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [14067.857432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14067.911807] alg: No test for adler32 (adler32-zlib) [14069.018633] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14069.348403] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [14071.056190] Key type lgssc registered [14072.501877] Lustre: Echo OBD driver; http://www.lustre.org/ [14118.435870] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14118.478648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14120.077559] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14120.155824] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14120.330666] Lustre: tst250fs-MDT0000: new disk, initializing [14120.484528] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14120.534916] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14126.140323] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14142.563803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14142.793934] Lustre: 402476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14142.948192] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14142.952914] Lustre: Skipped 1 previous similar message [14143.058263] Lustre: tst250fs-MDT0001: new disk, initializing [14143.270962] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14143.382665] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14143.418164] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14152.583866] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14158.891900] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14168.026924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14168.411965] Lustre: tst250fs-OST0000: new disk, initializing [14168.416287] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14168.425618] Lustre: 404110:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14168.556489] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14168.953080] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14168.962687] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14169.130718] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14176.231400] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14188.872517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14189.115726] Lustre: tst250fs-OST0001: new disk, initializing [14189.124671] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14189.133685] Lustre: 404983:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14189.304236] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14197.210717] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14197.908581] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14197.940957] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14198.114703] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14205.156559] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14207.841757] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14207.855833] Lustre: Skipped 1 previous similar message [14230.498178] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14230.515447] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14230.537554] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14234.084861] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14234.087984] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14237.000311] Lustre: server umount tst250fs-MDT0000 complete [14241.791271] LustreError: 403272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455362 with bad export cookie 15284986316992074363 [14241.794517] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14241.802889] LustreError: 403272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14242.143980] Lustre: server umount tst250fs-MDT0001 complete [14256.705760] Lustre: server umount tst250fs-OST0000 complete [14260.513019] Lustre: 400698:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455365/real 1788455365] req@ffff9fc234289f80 x1875331189384960/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455381 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14260.536956] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14260.549694] Lustre: Skipped 2 previous similar messages [14260.992766] Lustre: server umount tst250fs-OST0001 complete [14281.905342] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [14290.781075] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [14333.858915] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:10:51 (1788455451) [14336.779984] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14361.663860] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [14374.393290] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [14412.630763] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14120 sec ========= 13:12:10 (1788455530) [14415.293750] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:12:13 (1788455533) === [14419.739203] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:12:18 (1788455538) === [14472.240831] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [14475.340267] Key type lgssc unregistered [14475.626851] LNet: 414122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14475.635921] LNetError: 414122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14475.667938] LNet: Removed LNI 192.168.203.115@tcp [14476.597179] Key type .llcrypt unregistered [14476.602856] Key type ._llcrypt unregistered