[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 528684080 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002010] Switched APIC routing to physical x2apic. [ 0.003014] kvm-guest: setup PV IPIs [ 0.005767] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007013] pid_max: default: 32768 minimum: 301 [ 0.008147] LSM: Security Framework initializing [ 0.009063] Yama: becoming mindful. [ 0.010043] SELinux: Initializing. [ 0.011064] *** VALIDATE selinux *** [ 0.019720] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023696] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025123] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026124] *** VALIDATE tmpfs *** [ 0.027478] *** VALIDATE proc *** [ 0.029221] *** VALIDATE cgroup *** [ 0.030010] *** VALIDATE cgroup2 *** [ 0.031269] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033119] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035031] Spectre V2 : User space: Vulnerable [ 0.036010] Speculative Store Bypass: Vulnerable [ 0.039693] debug: unmapping init [mem 0xffffffffb0059000-0xffffffffb0060fff] [ 0.041852] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042745] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043035] ... version: 2 [ 0.044013] ... bit width: 48 [ 0.045012] ... generic registers: 4 [ 0.046015] ... value mask: 0000ffffffffffff [ 0.047017] ... max period: 00007fffffffffff [ 0.048017] ... fixed-purpose events: 3 [ 0.049016] ... event mask: 000000070000000f [ 0.050305] rcu: Hierarchical SRCU implementation. [ 0.052558] smp: Bringing up secondary CPUs ... [ 0.053634] x86: Booting SMP configuration: [ 0.054032] .... node #0, CPUs: #1 #2 #3 [ 0.057832] smp: Brought up 1 node, 4 CPUs [ 0.059015] smpboot: Max logical packages: 1 [ 0.060020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137410] node 0 deferred pages initialised in 75ms [ 0.141009] devtmpfs: initialized [ 0.142310] x86/mm: Memory block size: 128MB [ 0.144610] gcov: version magic: 0x41383552 [ 0.148287] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149186] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152333] pinctrl core: initialized pinctrl subsystem [ 0.154194] [ 0.154852] ************************************************************* [ 0.158013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160010] ** ** [ 0.163010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165011] ** ** [ 0.167013] ** This means that this kernel is built to expose internal ** [ 0.169014] ** IOMMU data structures, which may compromise security on ** [ 0.172014] ** your system. ** [ 0.174014] ** ** [ 0.176011] ** If you see this message and you are not debugging the ** [ 0.177011] ** kernel, report this immediately to your vendor! ** [ 0.179020] ** ** [ 0.181013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182011] ************************************************************* [ 0.184617] NET: Registered protocol family 16 [ 0.187526] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199038] cpuidle: using governor menu [ 0.200701] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204524] PCI: Using configuration type 1 for base access [ 0.206135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.216054] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.218061] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.222138] cryptd: max_cpu_qlen set to 1000 [ 0.225314] ACPI: Added _OSI(Module Device) [ 0.226020] ACPI: Added _OSI(Processor Device) [ 0.228013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233649] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.240285] ACPI: Interpreter enabled [ 0.241061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243012] ACPI: Using IOAPIC for interrupt routing [ 0.244160] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.247085] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.260791] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.264049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.267026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.272083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279409] acpiphp: Slot [2] registered [ 0.282138] acpiphp: Slot [5] registered [ 0.284131] acpiphp: Slot [6] registered [ 0.286122] acpiphp: Slot [7] registered [ 0.288128] acpiphp: Slot [8] registered [ 0.290149] acpiphp: Slot [9] registered [ 0.292121] acpiphp: Slot [10] registered [ 0.294134] acpiphp: Slot [3] registered [ 0.296111] acpiphp: Slot [4] registered [ 0.297100] acpiphp: Slot [11] registered [ 0.299140] acpiphp: Slot [12] registered [ 0.301108] acpiphp: Slot [13] registered [ 0.303116] acpiphp: Slot [14] registered [ 0.304093] acpiphp: Slot [15] registered [ 0.305102] acpiphp: Slot [16] registered [ 0.307080] acpiphp: Slot [17] registered [ 0.308080] acpiphp: Slot [18] registered [ 0.309105] acpiphp: Slot [19] registered [ 0.311091] acpiphp: Slot [20] registered [ 0.312069] acpiphp: Slot [21] registered [ 0.314066] acpiphp: Slot [22] registered [ 0.315010] acpiphp: Slot [23] registered [ 0.316086] acpiphp: Slot [24] registered [ 0.317108] acpiphp: Slot [25] registered [ 0.319115] acpiphp: Slot [26] registered [ 0.321166] acpiphp: Slot [27] registered [ 0.323111] acpiphp: Slot [28] registered [ 0.324073] acpiphp: Slot [29] registered [ 0.325128] acpiphp: Slot [30] registered [ 0.326102] acpiphp: Slot [31] registered [ 0.329085] PCI host bridge to bus 0000:00 [ 0.330017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.333026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.335026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.337031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.340022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.343024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345270] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.347823] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351515] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.364959] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.370097] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.375042] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.378021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.381509] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383831] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.386042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.390008] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.396018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.409018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.415016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.420097] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.426015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.433017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.449000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.460254] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.469018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.477019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.501019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.514218] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.521017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.528017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.549016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.559000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.568016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.575017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.592018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.608435] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.615020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.622019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.645020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.656235] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.671015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.681018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.718029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.733899] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.736365] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.739371] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.741332] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.744245] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.750056] iommu: Default domain type: Passthrough [ 0.751496] SCSI subsystem initialized [ 0.753097] ACPI: bus type USB registered [ 0.754099] usbcore: registered new interface driver usbfs [ 0.756104] usbcore: registered new interface driver hub [ 0.757077] usbcore: registered new device driver usb [ 0.759166] pps_core: LinuxPPS API ver. 1 registered [ 0.761011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.764073] PTP clock support registered [ 0.766157] EDAC MC: Ver: 3.0.0 [ 0.768519] PCI: Using ACPI for IRQ routing [ 0.770901] NetLabel: Initializing [ 0.772011] NetLabel: domain hash size = 128 [ 0.774009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.776107] NetLabel: unlabeled traffic allowed by default [ 0.777090] vgaarb: loaded [ 0.779256] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.781009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.785009] clocksource: Switched to clocksource kvm-clock [ 0.882220] VFS: Disk quotas dquot_6.6.0 [ 0.883535] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.885715] *** VALIDATE ramfs *** [ 0.886577] *** VALIDATE hugetlbfs *** [ 0.887924] pnp: PnP ACPI init [ 0.890514] pnp: PnP ACPI: found 6 devices [ 0.906312] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.909462] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.914351] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.916910] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.919769] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.921902] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.924554] NET: Registered protocol family 2 [ 0.926834] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.932567] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.935967] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.941562] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.945092] TCP: Hash tables configured (established 65536 bind 65536) [ 0.947714] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.949597] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.951933] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.954498] NET: Registered protocol family 1 [ 0.956437] RPC: Registered named UNIX socket transport module. [ 0.958415] RPC: Registered udp transport module. [ 0.960088] RPC: Registered tcp transport module. [ 0.962425] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.964695] NET: Registered protocol family 44 [ 0.966422] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.968632] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.970712] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.972837] PCI: CLS 0 bytes, default 64 [ 0.975526] Unpacking initramfs... [ 2.407695] debug: unmapping init [mem 0xffff9c633cc54000-0xffff9c633ffbffff] [ 2.412059] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.414860] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.418400] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.939745] Initialise system trusted keyrings [ 2.941632] Key type blacklist registered [ 2.944101] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.955529] zbud: loaded [ 2.959333] *** VALIDATE nfs *** [ 2.960708] *** VALIDATE nfs4 *** [ 2.962282] pstore: using deflate compression [ 2.967641] Platform Keyring initialized [ 3.095087] NET: Registered protocol family 38 [ 3.097078] Key type asymmetric registered [ 3.098820] Asymmetric key parser 'x509' registered [ 3.100719] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.104582] io scheduler mq-deadline registered [ 3.106458] io scheduler kyber registered [ 3.108248] io scheduler bfq registered [ 3.111327] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.114166] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.117150] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.120436] ACPI: Power Button [PWRF] [ 3.126080] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.134071] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.147828] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.160952] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.183041] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.215308] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.246602] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.252103] Non-volatile memory driver v1.3 [ 3.253910] Linux agpgart interface v0.103 [ 3.288855] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.291421] vda: detected capacity change from 0 to 76681216 [ 3.304965] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.307568] vdb: detected capacity change from 0 to 1073741824 [ 3.322940] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.325425] vdc: detected capacity change from 0 to 2621440000 [ 3.343850] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.346253] vdd: detected capacity change from 0 to 2621440000 [ 3.366617] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.369402] vde: detected capacity change from 0 to 4294967296 [ 3.390680] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.393605] vdf: detected capacity change from 0 to 4294967296 [ 3.406571] libphy: Fixed MDIO Bus: probed [ 3.412192] usbcore: registered new interface driver usbserial_generic [ 3.414062] usbserial: USB Serial support registered for generic [ 3.415977] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.420085] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.421668] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.424100] mousedev: PS/2 mouse device common for all mice [ 3.426603] rtc_cmos 00:05: RTC can wake from S4 [ 3.430374] rtc_cmos 00:05: registered as rtc0 [ 3.430813] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.434145] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.439826] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.440448] intel_pstate: CPU model not supported [ 3.444769] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.446502] hid: raw HID events driver (C) Jiri Kosina [ 3.450731] usbcore: registered new interface driver usbhid [ 3.454219] usbhid: USB HID core driver [ 3.455918] drop_monitor: Initializing network drop monitor service [ 3.458148] Initializing XFRM netlink socket [ 3.460464] NET: Registered protocol family 10 [ 3.463743] Segment Routing with IPv6 [ 3.465183] NET: Registered protocol family 17 [ 3.467324] mpls_gso: MPLS GSO support [ 3.473368] RAS: Correctable Errors collector initialized. [ 3.475138] AVX version of gcm_enc/dec engaged. [ 3.476251] AES CTR mode by8 optimization enabled [ 3.558068] sched_clock: Marking stable (3558035791, 0)->(4477867811, -919832020) [ 3.560865] registered taskstats version 1 [ 3.562543] Loading compiled-in X.509 certificates [ 3.564382] zswap: loaded using pool lzo/zbud [ 3.594333] Key type big_key registered [ 3.610336] Key type encrypted registered [ 3.611960] ima: No TPM chip found, activating TPM-bypass! [ 3.613917] ima: Allocated hash algorithm: sha1 [ 3.615706] ima: No architecture policies found [ 3.617530] evm: Initialising EVM extended attributes: [ 3.619318] evm: security.selinux [ 3.620633] evm: security.ima [ 3.621532] evm: security.capability [ 3.622621] evm: HMAC attrs: 0x1 [ 3.625558] rtc_cmos 00:05: setting system clock to 2026-09-03 23:10:46 UTC (1788477046) [ 3.631402] debug: unmapping init [mem 0xffffffffb1003000-0xffffffffb11fffff] [ 3.634180] debug: unmapping init [mem 0xffffffffafd82000-0xffffffffb0058fff] [ 3.642272] Write protecting the kernel read-only data: 28672k [ 3.645233] debug: unmapping init [mem 0xffffffffae403000-0xffffffffae5fffff] [ 3.647510] debug: unmapping init [mem 0xffffffffaed14000-0xffffffffaedfffff] [ 3.680802] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.688609] systemd[1]: Detected virtualization kvm. [ 3.691198] systemd[1]: Detected architecture x86-64. [ 3.692721] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.716208] systemd[1]: No hostname configured. [ 3.717740] systemd[1]: Set hostname to . [ 3.719412] random: systemd: uninitialized urandom read (16 bytes read) [ 3.721584] systemd[1]: Initializing machine ID from random generator. [ 3.777545] random: ln: uninitialized urandom read (6 bytes read) [ 3.857868] random: systemd: uninitialized urandom read (16 bytes read) [ 3.860595] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.864126] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.868461] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.612393] device-mapper: uevent: version 1.0.3 [ 4.614213] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.338116] random: fast init done [ 5.412244] virtio_net virtio0 ens2: renamed from eth0 [ 5.521175] scsi host0: ata_piix [ 5.537348] scsi host1: ata_piix [ 5.538945] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.543113] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.249711] dracut-initqueue[575]: RTNETLINK answers: File exists [ 10.127995] random: crng init done [ 10.129895] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.690648] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.885605] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.173539] SELinux: Disabled at runtime. [ 12.239955] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.248945] systemd[1]: Detected virtualization kvm. [ 12.251041] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.795726] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.799759] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.808421] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.813230] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.816065] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.823445] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.829629] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.886808] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.369855] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.712820] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.764585] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.889820] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.930828] EDAC sbridge: Ver: 1.1.2 [ 17.109264] Key type dns_resolver registered [ 17.941678] NFS: Registering the id_resolver key type [ 17.958691] Key type id_resolver registered [ 17.965265] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (5s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg412-server login: [ 79.408102] libcfs: loading out-of-tree module taints kernel. [ 79.442925] Key type ._llcrypt registered [ 79.444700] Key type .llcrypt registered [ 79.550358] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 102.329523] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 104.086709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 104.101455] alg: No test for adler32 (adler32-zlib) [ 106.051122] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 107.366718] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 109.223345] Key type lgssc registered [ 111.935464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.702093] hrtimer: interrupt took 4367350 ns [ 131.151051] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 180.027686] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 198.092155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 198.128179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.570784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 199.659850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 199.809824] Lustre: lustre-MDT0000: new disk, initializing [ 199.879649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.908415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 205.901436] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.272622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.425925] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 222.492208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 222.499019] Lustre: Skipped 1 previous similar message [ 222.571609] Lustre: lustre-MDT0001: new disk, initializing [ 222.628753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.670600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 222.678881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 228.613740] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.058509] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 246.253127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 246.577448] Lustre: lustre-OST0000: new disk, initializing [ 246.582359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 246.590343] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 246.717132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.964026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 252.979404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 253.068049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 254.391429] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 272.379341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 272.506523] Lustre: lustre-OST0001: new disk, initializing [ 272.509205] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 272.514275] Lustre: 9518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 272.568672] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 279.916586] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 280.094556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 280.105240] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 280.186056] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 294.129396] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 301.967996] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 309.328749] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing check_logdir /tmp/testlogs/ [ 314.908954] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing yml_node [ 320.166812] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 323.266081] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 327.100905] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 329.266452] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 19:16:09 EDT 2026 [ 350.334184] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 352.652951] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 371.173487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 371.183865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 371.198605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 372.194146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 372.196743] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.203430] Lustre: Skipped 1 previous similar message [ 372.215428] Lustre: Skipped 2 previous similar messages [ 376.086587] Lustre: server umount lustre-MDT0000 complete [ 381.155153] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477424 with bad export cookie 12190483990987584968 [ 381.164294] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 381.166096] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 382.432535] LustreError: 6513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 382.433614] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.433882] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 382.433886] Lustre: Skipped 1 previous similar message [ 382.466737] LustreError: 6513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 382.502353] Lustre: Skipped 1 previous similar message [ 386.480876] LustreError: 6512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.481616] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 386.542037] Lustre: Skipped 2 previous similar messages [ 387.721440] Lustre: server umount lustre-MDT0001 complete [ 392.745199] Lustre: server umount lustre-OST0000 complete [ 397.789152] Lustre: server umount lustre-OST0001 complete [ 406.991318] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 418.861491] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 463.942808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 464.227743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 464.309827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 464.454704] Lustre: lustre-MDT0000: new disk, initializing [ 464.605625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.635677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 470.091491] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 483.513473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 483.698720] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 483.844852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 483.850421] Lustre: Skipped 1 previous similar message [ 483.926536] Lustre: lustre-MDT0001: new disk, initializing [ 484.037111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 484.089307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 484.106641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 490.778822] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 502.459371] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 509.249642] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 517.479930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 517.843746] Lustre: lustre-OST0000: new disk, initializing [ 517.847957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 517.854486] Lustre: 17051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 517.968617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 519.736912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 519.765575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 519.996676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 525.579116] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 536.611707] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 542.418548] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 542.784146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 549.166605] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 549.436613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 556.001859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.027853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 556.066942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 556.080341] Lustre: Skipped 1 previous similar message [ 566.239655] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 566.252611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 566.268835] Lustre: Skipped 3 previous similar messages [ 566.526477] Lustre: server umount lustre-OST0000 complete [ 571.360991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.383741] Lustre: Skipped 1 previous similar message [ 577.564085] Lustre: server umount lustre-MDT0000 complete [ 581.603382] LustreError: 15093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.643868] LustreError: 15093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 582.951941] LustreError: 17061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477625 with bad export cookie 12190483990987586501 [ 582.955059] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 582.967265] LustreError: 17061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 583.480495] Lustre: server umount lustre-MDT0001 complete [ 590.574920] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 19:20:31 (1788477631) [ 615.106677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 615.657261] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 615.771149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 621.036647] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.170325] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 631.265960] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 631.292875] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 631.590740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.044427] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 646.166473] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 654.199756] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 657.086961] Lustre: Failing over lustre-MDT0000 [ 657.576854] Lustre: server umount lustre-MDT0000 complete [ 657.888071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 657.900974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.925026] Lustre: Skipped 1 previous similar message [ 657.938141] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.343126] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788477704/real 1788477704] req@ffff9c6286808700 x1875354216213760/t0(0) o400->MGC192.168.204.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788477720 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 677.370383] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 678.608479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.585316] LustreError: 3635:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9c6286f56300 x1875354216216448/t0(0) o250->MGC192.168.204.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 688.063831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 688.072892] Lustre: Skipped 1 previous similar message [ 688.160426] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 693.220984] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 693.231771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 693.273767] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 693.571161] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 705.318948] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 711.660716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 711.667571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 711.675893] Lustre: Skipped 2 previous similar messages [ 711.691089] Lustre: Skipped 5 previous similar messages [ 716.007243] Lustre: server umount lustre-MDT0000 complete [ 716.770374] LustreError: 19923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.820428] LustreError: 19923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 722.334964] LustreError: 19910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477765 with bad export cookie 12190483990987587915 [ 722.348018] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.355876] LustreError: 19910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 722.949960] Lustre: server umount lustre-MDT0001 complete [ 734.581038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 741.246740] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 752.237974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.582823] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 767.892174] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 774.993957] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 777.500178] Lustre: Failing over lustre-MDT0000 [ 777.697788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 777.733648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 777.747172] Lustre: Skipped 1 previous similar message [ 777.989416] Lustre: server umount lustre-MDT0000 complete [ 783.842376] LustreError: 24000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 783.877204] LustreError: 24000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 795.103116] Lustre: 3637:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788477821/real 1788477821] req@ffff9c63bcf38700 x1875354216245888/t0(0) o400->MGC192.168.204.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788477837 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 795.152826] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 799.917239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 806.726419] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 812.008471] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 812.015369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 812.033594] Lustre: Skipped 1 previous similar message [ 812.049652] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 812.277675] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 824.086609] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 834.825325] Lustre: server umount lustre-MDT0000 complete [ 839.717977] LustreError: 23252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477882 with bad export cookie 12190483990987588853 [ 839.731336] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 839.737698] LustreError: 23252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 840.126504] Lustre: server umount lustre-MDT0001 complete [ 851.373857] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 19:24:51 (1788477891) [ 874.247343] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 885.670910] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 932.749707] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 946.848241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.211516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 947.258335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 947.404923] Lustre: lustre-MDT0000: new disk, initializing [ 947.507146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 947.513739] Lustre: Skipped 3 previous similar messages [ 947.539734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 953.282255] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 966.029620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 966.315225] Lustre: 29649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 966.592358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 972.495994] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 984.504131] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 991.357642] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 999.405845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 999.810969] Lustre: 31623:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1001.385992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1001.460485] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1001.514812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1001.525952] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1007.668667] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1017.788666] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1024.048772] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1047.532036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1047.540815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.547340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1047.562905] Lustre: Skipped 4 previous similar messages [ 1047.570470] Lustre: Skipped 5 previous similar messages [ 1059.295133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1059.711583] Lustre: server umount lustre-OST0000 complete [ 1070.692613] Lustre: server umount lustre-MDT0000 complete [ 1073.121289] LustreError: 30886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.153470] LustreError: 30886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 1075.185730] LustreError: 29644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478118 with bad export cookie 12190483990987589259 [ 1075.190631] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1075.199865] LustreError: 29644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1075.679879] Lustre: server umount lustre-MDT0001 complete [ 1088.249593] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 1091.988207] Key type lgssc unregistered [ 1092.329722] LNet: 33600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1092.341518] LNetError: 33600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1092.354669] LNet: Removed LNI 192.168.204.112@tcp [ 1093.588299] Key type .llcrypt unregistered [ 1093.593559] Key type ._llcrypt unregistered [ 1122.322345] Key type ._llcrypt registered [ 1122.325295] Key type .llcrypt registered [ 1122.424556] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 1123.457448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1123.616311] alg: No test for adler32 (adler32-zlib) [ 1124.780965] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1125.048490] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 1126.743251] Key type lgssc registered [ 1127.841932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1136.755155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1136.818690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1138.389768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1143.687884] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1148.903098] LustreError: 35012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1148.924852] LustreError: 35012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1154.023460] LustreError: 35011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1154.092746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.694219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1160.500046] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1170.610625] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1177.942495] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1185.826887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1186.113480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.149771] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1199.496114] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1202.954443] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1208.753518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1224.762677] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 19:31:05 (1788478265) [ 1227.428770] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1229.589246] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1237.652738] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 19:31:18 (1788478278) [ 1247.227454] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 19:31:27 (1788478287) [ 1248.394198] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1257.731931] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1257.736653] Lustre: Skipped 2 previous similar messages [ 1259.776536] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 19:31:40 (1788478300) [ 1261.862579] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1261.872530] Lustre: Skipped 2 previous similar messages [ 1271.049959] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1271.055852] Lustre: Skipped 4 previous similar messages [ 1286.447832] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 19:32:07 (1788478327) [ 1289.573194] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1293.402679] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1302.234337] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 19:32:23 (1788478343) [ 1304.187910] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1304.200184] Lustre: Skipped 6 previous similar messages [ 1318.983657] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 19:32:39 (1788478359) [ 1336.800452] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1336.804362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.817072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1336.833041] Lustre: Skipped 1 previous similar message [ 1340.211897] Lustre: server umount lustre-OST0000 complete [ 1343.976988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1343.990792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1344.020507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1344.030213] Lustre: Skipped 1 previous similar message [ 1347.041522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1347.047243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.165693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1352.182430] Lustre: Skipped 2 previous similar messages [ 1356.257281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1356.271646] Lustre: Skipped 1 previous similar message [ 1358.304213] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1358.629703] Lustre: server umount lustre-MDT0000 complete [ 1361.385963] LustreError: 35006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.409238] LustreError: 35006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1362.936845] LustreError: 34994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478405 with bad export cookie 333805485480839061 [ 1362.937687] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1362.961624] LustreError: 34994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1363.411817] Lustre: server umount lustre-MDT0001 complete [ 1374.981785] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 1378.134890] Key type lgssc unregistered [ 1378.435081] LNet: 42876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.442385] LNetError: 42876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1378.462808] LNet: Removed LNI 192.168.204.112@tcp [ 1379.622216] Key type .llcrypt unregistered [ 1379.624961] Key type ._llcrypt unregistered [ 1403.423297] Key type ._llcrypt registered [ 1403.426423] Key type .llcrypt registered [ 1403.614359] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 1421.605573] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 1422.759491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1423.001562] alg: No test for adler32 (adler32-zlib) [ 1424.106640] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1424.562972] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 1426.399714] Key type lgssc registered [ 1427.425694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1470.964519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1471.002077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1472.407738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1472.457995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1472.563275] Lustre: lustre-MDT0000: new disk, initializing [ 1472.676335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.714547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1478.031347] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1489.649434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.767289] Lustre: 46028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1489.853720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1489.864629] Lustre: Skipped 1 previous similar message [ 1489.970910] Lustre: lustre-MDT0001: new disk, initializing [ 1490.053072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1490.082323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1490.090229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1494.260696] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1503.793964] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1511.393181] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1521.468245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1522.019409] Lustre: lustre-OST0000: new disk, initializing [ 1522.034488] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1522.055602] Lustre: 47997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1522.250384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1527.383505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1527.403905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1527.585791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1531.051116] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1541.820072] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1548.205802] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1548.511468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1554.288784] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1554.509313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1558.496486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1558.498772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1558.506805] LustreError: Skipped 1 previous similar message [ 1558.509513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1558.529108] Lustre: Skipped 1 previous similar message [ 1563.625776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1563.640114] Lustre: Skipped 2 previous similar messages [ 1568.740232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1568.751348] Lustre: Skipped 1 previous similar message [ 1571.295150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1571.595718] Lustre: server umount lustre-OST0000 complete [ 1577.443262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1577.451454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1577.474260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1578.977181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.720953] Lustre: server umount lustre-MDT0000 complete [ 1585.545987] LustreError: 46020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478628 with bad export cookie 11616229842531605018 [ 1585.548474] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1585.560038] LustreError: 46020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1585.921247] Lustre: server umount lustre-MDT0001 complete [ 1596.415929] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 19:37:17 (1788478637) [ 1603.816456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.451978] LustreError: 50458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1604.464706] LustreError: 50458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1604.574066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1609.700516] LustreError: 50459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1609.765819] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1614.817559] LustreError: 50458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1619.319851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1619.695230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1625.460291] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1633.356350] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1639.154432] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1646.592365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1647.044074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1653.970169] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1661.969756] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1670.107305] LustreError: 52990:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1671.484540] Lustre: Modifying parameter general.timeout=1 in log params [ 1672.917527] Lustre: Modifying parameter general.timeout=2 in log params [ 1675.216565] Lustre: Modifying parameter general.timeout=4 in log params [ 1675.221365] Lustre: Skipped 1 previous similar message [ 1679.608330] Lustre: Modifying parameter general.timeout=8 in log params [ 1679.623037] Lustre: Skipped 3 previous similar messages [ 1688.048333] Lustre: Modifying parameter general.timeout=15 in log params [ 1688.050942] Lustre: Skipped 6 previous similar messages [ 1704.252500] Lustre: Modifying parameter general.timeout=31 in log params [ 1704.261081] Lustre: Skipped 15 previous similar messages [ 1737.310058] Lustre: Modifying parameter general.timeout=64 in log params [ 1737.312338] Lustre: Skipped 32 previous similar messages [ 1802.025615] Lustre: Modifying parameter general.timeout=125 in log params [ 1802.035159] Lustre: Skipped 60 previous similar messages [ 1818.643557] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 19:40:59 (1788478859) [ 1822.083519] Lustre: 59950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1824.131877] Lustre: 60048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1826.163956] Lustre: 60146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1830.565600] Lustre: 60359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1830.582615] Lustre: 60359:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1864.435128] Lustre: 61819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1864.441143] Lustre: 61819:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1864.463441] LustreError: 61819:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1869.501430] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 19:41:49 (1788478909) [ 1885.156412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.160629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1885.177437] Lustre: Skipped 1 previous similar message [ 1885.215710] Lustre: Skipped 4 previous similar messages [ 1888.759381] Lustre: server umount lustre-MDT0000 complete [ 1890.273662] LustreError: 50453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1890.295865] LustreError: 50453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1892.620950] LustreError: 50440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478935 with bad export cookie 11616229842531605900 [ 1892.623960] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1892.643093] LustreError: 50440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1893.143358] Lustre: server umount lustre-MDT0001 complete [ 1907.758167] Lustre: server umount lustre-OST0000 complete [ 1938.080956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.626093] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1938.727803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1938.746551] Lustre: Skipped 27 previous similar messages [ 1939.157581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.306679] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1957.958607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.089334] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1958.128220] Lustre: 64351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1964.480648] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1974.382403] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1981.421693] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1988.721311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1989.026589] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1989.398802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.406581] Lustre: Skipped 1 previous similar message [ 1996.426615] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2004.113540] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2005.933251] Lustre: 66918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2012.643481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.657516] Lustre: Skipped 1 previous similar message [ 2012.666126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2017.249139] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2018.742986] Lustre: server umount lustre-OST0000 complete [ 2022.888265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.899652] Lustre: Skipped 2 previous similar messages [ 2033.124203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2033.132767] Lustre: Skipped 7 previous similar messages [ 2037.215166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2037.428949] Lustre: server umount lustre-MDT0000 complete [ 2041.446715] LustreError: 64341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479084 with bad export cookie 11616229842531609106 [ 2041.456197] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2041.457410] LustreError: 64341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2041.928608] Lustre: server umount lustre-MDT0001 complete [ 2053.833869] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 2057.321623] Key type lgssc unregistered [ 2057.736795] LNet: 68127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2057.743871] LNetError: 68127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2057.787190] LNet: Removed LNI 192.168.204.112@tcp [ 2058.777565] Key type .llcrypt unregistered [ 2058.781504] Key type ._llcrypt unregistered [ 2085.572839] Key type ._llcrypt registered [ 2085.580848] Key type .llcrypt registered [ 2085.700236] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 2086.976565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2087.005037] alg: No test for adler32 (adler32-zlib) [ 2088.100593] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2088.446734] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 2090.143188] Key type lgssc registered [ 2091.442643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2100.445118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2100.467038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2102.171900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2106.919566] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2111.480187] LustreError: 69538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2111.511743] LustreError: 69538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2116.049993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.676511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2121.343487] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2129.772761] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2135.758400] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2143.793595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2144.536263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.688575] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2160.095199] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2171.309993] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2182.639636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.647586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2182.670345] Lustre: Skipped 2 previous similar messages [ 2182.685843] Lustre: Skipped 2 previous similar messages [ 2187.574342] Lustre: server umount lustre-MDT0000 complete [ 2187.750431] LustreError: 69539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.786103] LustreError: 69539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2191.811180] LustreError: 69519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479234 with bad export cookie 1204513777706056926 [ 2191.821273] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2191.825470] LustreError: 69519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2192.155367] Lustre: server umount lustre-MDT0001 complete [ 2206.961451] Lustre: server umount lustre-OST0000 complete [ 2240.391400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.781252] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2240.823796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2241.147777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2247.076849] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2258.940513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.091747] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2259.164908] Lustre: 74410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2259.858490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2265.254973] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2275.983729] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2283.685807] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2291.125737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2291.358781] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2291.860385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2299.225746] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2310.059385] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2312.406042] Lustre: 76991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2314.896240] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2323.426775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2323.444985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.463091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.943083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2328.121428] Lustre: server umount lustre-OST0000 complete [ 2334.176627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.197784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.212027] Lustre: Skipped 2 previous similar messages [ 2334.245072] Lustre: Skipped 2 previous similar messages [ 2339.298961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2339.308752] Lustre: Skipped 1 previous similar message [ 2340.072791] Lustre: server umount lustre-MDT0000 complete [ 2343.396535] LustreError: 74422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.423425] LustreError: 74422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2344.732460] LustreError: 74401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479387 with bad export cookie 1204513777706057773 [ 2344.734038] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2344.756929] LustreError: 74401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2345.073982] Lustre: server umount lustre-MDT0001 complete [ 2360.097065] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 2363.749562] Key type lgssc unregistered [ 2364.135891] LNet: 78203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2364.142759] LNetError: 78203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2364.159851] LNet: Removed LNI 192.168.204.112@tcp [ 2365.224171] Key type .llcrypt unregistered [ 2365.228943] Key type ._llcrypt unregistered [ 2396.087706] Key type ._llcrypt registered [ 2396.090980] Key type .llcrypt registered [ 2396.261330] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 2397.794693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2398.085317] alg: No test for adler32 (adler32-zlib) [ 2399.377137] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2399.798239] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 2401.639220] Key type lgssc registered [ 2403.185127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2413.145598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2413.186329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2414.883346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2420.529579] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2424.297256] LustreError: 79616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.310896] LustreError: 79616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2429.410825] LustreError: 79615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.023897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.853536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2438.287510] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2447.675826] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2455.249556] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2463.048692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2463.656190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2471.279498] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2480.811748] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2492.283651] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2493.447453] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2494.792335] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2497.221488] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2497.231172] Lustre: Skipped 1 previous similar message [ 2501.929511] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2501.934203] Lustre: Skipped 4 previous similar messages [ 2510.412580] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2510.417339] Lustre: Skipped 8 previous similar messages [ 2526.996382] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2527.003404] Lustre: Skipped 13 previous similar messages [ 2559.591579] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2559.596225] Lustre: Skipped 29 previous similar messages [ 2604.600388] Lustre: 87723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2608.110778] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:54:08 (1788479648) [ 2609.975540] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2612.681247] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:54:12 (1788479652) [ 2620.390343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2620.405377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.435057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2622.438701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2622.441034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2622.442946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2622.461571] Lustre: Skipped 1 previous similar message [ 2624.642270] Lustre: server umount lustre-OST0000 complete [ 2631.140594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2631.149994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.174402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2635.967548] Lustre: server umount lustre-MDT0000 complete [ 2637.795558] LustreError: 81680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.822301] LustreError: 81680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2640.577774] LustreError: 79598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479683 with bad export cookie 3600868337262691094 [ 2640.580223] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2640.592620] LustreError: 79598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2641.147584] Lustre: server umount lustre-MDT0001 complete [ 2654.992599] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 2658.746209] Key type lgssc unregistered [ 2659.253968] LNet: 88972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2659.263507] LNetError: 88972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2659.287210] LNet: Removed LNI 192.168.204.112@tcp [ 2660.324851] Key type .llcrypt unregistered [ 2660.327262] Key type ._llcrypt unregistered [ 2665.760764] Key type ._llcrypt registered [ 2665.766703] Key type .llcrypt registered [ 2665.961217] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2671.205422] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules [ 2672.433989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2672.490807] alg: No test for adler32 (adler32-zlib) [ 2673.744146] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2674.106350] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 2675.872132] Key type lgssc registered [ 2677.884765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2679.855746] LustreError: 90217:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2699.969495] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 2702.788868] LustreError: 90217:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2711.478865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2711.493818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.151443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2719.699962] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2722.290742] LustreError: 91247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.338818] LustreError: 91247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2727.394139] LustreError: 91248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.171790] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:56:11 (1788479771) [ 2732.524124] LustreError: 91247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2737.271350] LustreError: 91279:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2737.286872] LustreError: 91279:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2737.296850] LustreError: 91279:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 2737.771175] Lustre: server umount lustre-MDT0000 complete [ 2756.024376] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 2760.442842] Key type lgssc unregistered [ 2760.910460] LNet: 92776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2760.919282] LNetError: 92776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2760.940425] LNet: Removed LNI 192.168.204.112@tcp [ 2762.093520] Key type .llcrypt unregistered [ 2762.098454] Key type ._llcrypt unregistered [ 2783.733412] Key type ._llcrypt registered [ 2783.736367] Key type .llcrypt registered [ 2783.887288] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 2785.108976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2785.293897] alg: No test for adler32 (adler32-zlib) [ 2786.353744] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2786.686967] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 2788.431859] Key type lgssc registered [ 2789.417865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2798.443281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2798.482383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.343928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2805.861995] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2810.851662] LustreError: 93989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.873024] LustreError: 93989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2815.972181] LustreError: 93990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.270834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.983777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2821.808499] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2830.543232] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2838.614675] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2846.345353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2847.122625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2854.185176] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2862.724943] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2926.054265] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:59:26 (1788479966) [ 2928.612268] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2931.184607] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:59:31 (1788479971) [ 2937.327056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2937.335331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2937.365350] Lustre: Skipped 1 previous similar message [ 2937.378838] Lustre: Skipped 2 previous similar messages [ 2942.441808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.448122] Lustre: Skipped 2 previous similar messages [ 2942.883435] Lustre: server umount lustre-MDT0000 complete [ 2947.554306] LustreError: 93984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.586672] LustreError: 93984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2948.133609] LustreError: 93970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479991 with bad export cookie 10717078611536017614 [ 2948.155498] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2948.161893] LustreError: 93970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2948.682743] Lustre: server umount lustre-MDT0001 complete [ 2963.131409] Lustre: server umount lustre-OST0000 complete [ 2974.846664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.216650] LustreError: 98309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.351254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2980.273355] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2980.834643] LustreError: 98310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.591428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2991.197348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2996.319508] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3004.368363] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3011.324614] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3027.761082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3028.023094] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3028.043366] LustreError: 98297:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3028.081458] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3028.104441] LustreError: 100407:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 3028.110981] LustreError: 100407:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 3028.120933] LustreError: 100407:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3028.239520] Lustre: server umount lustre-OST0000 complete [ 3028.241655] LustreError: 100407:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3037.932816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3038.146586] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3038.154634] LustreError: 98299:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3038.187083] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3038.198358] LustreError: 100874:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 3038.208696] LustreError: 100874:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 3038.222522] LustreError: 100874:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3038.328178] Lustre: server umount lustre-OST0000 complete [ 3038.330928] LustreError: 100874:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3049.370953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3049.767582] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3049.780165] Lustre: Found index 0 for lustre-OST0000, updating log [ 3049.784605] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3049.913433] Lustre: lustre-OST0000: new disk, initializing [ 3049.916767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3049.926816] Lustre: 101410:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3050.244303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.505375] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3064.412097] Lustre: Failing over lustre-OST0000 [ 3064.694274] Lustre: server umount lustre-OST0000 complete [ 3070.946456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.948965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3070.967084] Lustre: Skipped 2 previous similar messages [ 3070.985144] Lustre: Skipped 1 previous similar message [ 3074.576237] Lustre: server umount lustre-MDT0000 complete [ 3076.066854] LustreError: 98305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3076.084673] LustreError: 98305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3079.224339] LustreError: 100427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480122 with bad export cookie 10717078611536020358 [ 3079.237076] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3079.242805] LustreError: 100427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3090.098634] Lustre: server umount lustre-MDT0001 complete [ 3100.467839] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 20:02:21 (1788480141) [ 3117.090606] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3129.541711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3129.953141] LustreError: 103758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3130.093926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3135.384901] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3147.341366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.701068] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3159.074857] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3165.949373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3166.781712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3166.786322] Lustre: Skipped 1 previous similar message [ 3166.802135] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3172.904106] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3172.947139] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3172.947644] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3174.465421] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3185.039091] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3185.139035] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3185.258456] Lustre: lustre-OST0001: new disk, initializing [ 3185.264732] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3185.278794] Lustre: 106077:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3193.114759] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3195.515292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3195.530112] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3195.676823] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3201.527083] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3214.117261] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3219.958653] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3219.982279] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.003971] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3225.394422] Lustre: server umount lustre-MDT0001 complete [ 3226.471763] LustreError: 103754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3226.502559] LustreError: 103754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3235.597906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.719328] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3235.728737] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3235.794524] Lustre: 103748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3236.584214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3236.588696] Lustre: Skipped 1 previous similar message [ 3237.426954] LustreError: 107624:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3237.436408] LustreError: 107624:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3237.444708] LustreError: 103750:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3237.526680] LustreError: 107632:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.112@tcp: cfg command failed: rc = -17 [ 3237.538320] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3237.538320] [ 3241.206094] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3241.970417] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3251.408185] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 20:04:51 (1788480291) [ 3276.008232] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3295.560646] Lustre: 109709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3322.235283] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3326.007847] Lustre: 110844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3420.641191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3420.649193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.668552] Lustre: Skipped 2 previous similar messages [ 3420.677079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3420.695503] Lustre: Skipped 4 previous similar messages [ 3423.646944] Lustre: server umount lustre-MDT0000 complete [ 3426.277995] LustreError: 103758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.314889] LustreError: 103758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 3428.818291] LustreError: 103739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480471 with bad export cookie 10717078611536021359 [ 3428.830138] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3428.855629] LustreError: 103739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3429.285932] Lustre: server umount lustre-MDT0001 complete [ 3444.713205] Lustre: server umount lustre-OST0000 complete [ 3447.777145] Lustre: 93302:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788480474/real 1788480474] req@ffff9c6293da5f80 x1875357025547264/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788480490 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3447.811430] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3447.823963] Lustre: Skipped 3 previous similar messages [ 3449.824899] Lustre: 93299:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788480476/real 1788480476] req@ffff9c628b89f800 x1875357025547520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788480492 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3450.198690] Lustre: server umount lustre-OST0001 complete [ 3457.560544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3468.495406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3478.967331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3490.597281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3512.766215] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3525.266685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.301838] Lustre: lustre-MDT0000: reset Object Index mappings [ 3525.935083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3530.898426] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3539.705491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.741147] Lustre: lustre-MDT0001: reset Object Index mappings [ 3545.796487] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3549.797608] Lustre: 115939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3557.726840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3565.886717] LustreError: 116295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3565.894027] LustreError: 116295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3565.900663] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3566.185955] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3571.068850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3576.763442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3582.458478] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3582.463869] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3583.232768] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3590.459592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3593.875793] Lustre: 117819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3651.929339] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 20:11:32 (1788480692) [ 3677.673398] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.674757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3677.695607] Lustre: Skipped 2 previous similar messages [ 3677.703750] Lustre: Skipped 6 previous similar messages [ 3683.143871] Lustre: server umount lustre-MDT0000 complete [ 3687.562083] LustreError: 114773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480730 with bad export cookie 10717078611536068189 [ 3687.586870] LustreError: 114773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3687.588691] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3694.227435] Lustre: server umount lustre-MDT0001 complete [ 3706.271388] Lustre: server umount lustre-OST0000 complete [ 3716.566495] Lustre: server umount lustre-OST0001 complete [ 3723.909678] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 3734.359346] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3777.769755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3778.088776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3778.130612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3778.249337] Lustre: lustre-MDT0000: new disk, initializing [ 3778.399962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3784.186383] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3795.322671] Lustre: server umount lustre-MDT0000 complete [ 3808.572638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.185239] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3809.215764] Lustre: 122094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3809.239926] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3809.620549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.629242] Lustre: Skipped 4 previous similar messages [ 3816.285856] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3823.397825] Lustre: server umount lustre-MDT0000 complete [ 3833.401593] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 20:14:33 (1788480873) [ 3835.090561] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3838.002266] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 20:14:37 (1788480877) [ 3859.560465] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3871.858927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3877.900920] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3888.935755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3889.141484] Lustre: 124149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3889.203622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3889.208422] Lustre: Skipped 1 previous similar message [ 3889.380649] Lustre: lustre-MDT0001: new disk, initializing [ 3889.654726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3889.663902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3894.727961] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3899.617410] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3906.483307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3906.782417] Lustre: lustre-OST0000: new disk, initializing [ 3906.786553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3906.792103] Lustre: 125726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3908.095907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3908.113473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3908.186586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3913.112774] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3924.054583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3924.238553] Lustre: lustre-OST0001: new disk, initializing [ 3924.245780] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3924.256371] Lustre: 126596:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3925.785125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3925.795030] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3925.919064] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3930.717409] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3940.591343] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3944.431583] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4432.536957] Lustre: 133024:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4432.542482] Lustre: 133024:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4432.552568] Lustre: 133024:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4432.565057] Lustre: 133024:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4432.577635] Lustre: 133024:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4432.585213] Lustre: 133024:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4760.530194] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:29:59 (1788481799) [ 4778.982215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4778.992606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4778.998159] Lustre: Skipped 5 previous similar messages [ 4779.007529] Lustre: Skipped 9 previous similar messages [ 4783.039607] Lustre: server umount lustre-MDT0000 complete [ 4784.098087] LustreError: 124930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4784.117460] LustreError: 124930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 4787.036685] LustreError: 124139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788481829 with bad export cookie 10717078611536137391 [ 4787.052060] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4787.057377] LustreError: 124139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4789.216567] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4789.224949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4789.233982] Lustre: Skipped 2 previous similar messages [ 4793.602872] Lustre: server umount lustre-MDT0001 complete [ 4803.845383] Lustre: server umount lustre-OST0000 complete [ 4814.626183] Lustre: server umount lustre-OST0001 complete [ 4822.352331] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 4835.883455] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 4879.536052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4879.915389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4879.991321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4880.154901] Lustre: lustre-MDT0000: new disk, initializing [ 4880.288959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4880.294296] Lustre: Skipped 4 previous similar messages [ 4880.323322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4886.199801] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4898.322348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4898.534931] Lustre: 136086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4898.635333] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4898.644713] Lustre: Skipped 1 previous similar message [ 4898.773488] Lustre: lustre-MDT0001: new disk, initializing [ 4898.984426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4899.003916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4904.109915] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4914.547045] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4921.163158] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4928.926727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4929.513447] Lustre: lustre-OST0000: new disk, initializing [ 4929.531317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4929.542624] Lustre: 138054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4931.603063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4931.627279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4931.811790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4937.634662] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4947.736692] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4957.963218] Lustre: lustre-MDD0000: changelog on [ 4960.522962] Lustre: lustre-MDD0001: changelog on [ 7213.455443] Lustre: lustre-MDD0001: changelog off [ 7216.405842] Lustre: 136964:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 7217.523430] Lustre: lustre-MDD0000: changelog off [ 7225.314076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7225.326556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7225.351832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7225.368761] Lustre: Skipped 3 previous similar messages [ 7225.830203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7225.846676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7230.951724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7230.955786] Lustre: Skipped 2 previous similar messages [ 7231.330981] Lustre: server umount lustre-OST0000 complete [ 7236.065023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.081187] Lustre: Skipped 1 previous similar message [ 7236.087173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7236.102532] Lustre: Skipped 1 previous similar message [ 7241.185237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7241.198501] Lustre: Skipped 1 previous similar message [ 7242.114311] Lustre: server umount lustre-MDT0000 complete [ 7246.337842] LustreError: 139137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.350881] LustreError: 139137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7246.475870] LustreError: 136081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788484289 with bad export cookie 10717078611536964868 [ 7246.504943] LustreError: 136081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7246.505604] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7247.116426] Lustre: server umount lustre-MDT0001 complete [ 7262.485769] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 7266.030549] Key type lgssc unregistered [ 7266.334862] LNet: 147277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7266.349059] LNetError: 147277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7266.377636] LNet: Removed LNI 192.168.204.112@tcp [ 7267.284188] Key type .llcrypt unregistered [ 7267.286626] Key type ._llcrypt unregistered [ 7291.321548] Key type ._llcrypt registered [ 7291.325806] Key type .llcrypt registered [ 7291.445545] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 7315.083710] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 7316.199595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7316.267286] alg: No test for adler32 (adler32-zlib) [ 7317.665667] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 7318.139730] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 7319.775199] Key type lgssc registered [ 7321.337279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7362.098556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7362.127653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7363.430439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7363.464432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7363.554754] Lustre: lustre-MDT0000: new disk, initializing [ 7363.692238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7363.722657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7368.438322] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7379.903854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.119709] Lustre: 150480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7380.212116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7380.221255] Lustre: Skipped 1 previous similar message [ 7380.374564] Lustre: lustre-MDT0001: new disk, initializing [ 7380.533500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7380.582460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7380.597287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7386.182904] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7397.143667] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7404.081661] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7413.078249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7413.535760] Lustre: lustre-OST0000: new disk, initializing [ 7413.543157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7413.549826] Lustre: 152450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7413.676474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7420.123137] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7423.039839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7423.058481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7423.217663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7430.779909] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7437.548343] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_module obdecho/obdecho [ 7439.757281] LustreError: 153451:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7439.766425] LustreError: 153451:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7441.727786] LustreError: 153549:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7454.176913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7454.181828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7454.200742] Lustre: Skipped 2 previous similar messages [ 7454.225347] Lustre: Skipped 2 previous similar messages [ 7459.300516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7459.306743] Lustre: Skipped 2 previous similar messages [ 7460.173582] Lustre: server umount lustre-MDT0000 complete [ 7463.791794] LustreError: 151289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788484506 with bad export cookie 17526106635861449834 [ 7463.794159] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7463.809787] LustreError: 151289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7464.177566] Lustre: server umount lustre-MDT0001 complete [ 7477.396951] Lustre: server umount lustre-OST0000 complete [ 7500.071868] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 7513.215976] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 7554.921098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7555.197329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7555.273248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7555.375721] Lustre: lustre-MDT0000: new disk, initializing [ 7555.485043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7555.509192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7561.588885] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7574.356352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7574.503807] Lustre: 157264:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7574.570177] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7574.576581] Lustre: Skipped 1 previous similar message [ 7574.687238] Lustre: lustre-MDT0001: new disk, initializing [ 7574.696432] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7574.763424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7574.817751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7574.828597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7580.107030] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7591.483930] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7598.120551] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7607.549682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7607.897176] Lustre: lustre-OST0000: new disk, initializing [ 7607.905680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7607.918830] Lustre: 159237:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7608.082143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7614.060423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7614.081016] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7614.233302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7615.746664] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7626.200256] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7631.816952] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7632.136740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7637.441531] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7637.661308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7640.032153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7640.037292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7640.049788] LustreError: Skipped 1 previous similar message [ 7640.053445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7640.058725] Lustre: Skipped 1 previous similar message [ 7640.076760] Lustre: Skipped 1 previous similar message [ 7645.171056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7645.173625] Lustre: Skipped 1 previous similar message [ 7645.749151] Lustre: server umount lustre-OST0000 complete [ 7651.810371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7651.824277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.395774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.399468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7655.444113] Lustre: Skipped 2 previous similar messages [ 7657.425616] Lustre: server umount lustre-MDT0000 complete [ 7662.041738] LustreError: 157256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788484704 with bad export cookie 17526106635861450793 [ 7662.059964] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7662.060583] LustreError: 157256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7662.533789] Lustre: server umount lustre-MDT0001 complete [ 7667.201461] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:18:27 (1788484707) [ 7670.256312] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7672.257698] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:18:32 (1788484712) [ 7681.957328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.395882] LustreError: 161648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7682.425607] LustreError: 161648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7682.537642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7687.416926] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7687.677866] LustreError: 161648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7692.783242] LustreError: 161649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7696.941935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7703.180178] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7713.091684] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7719.693548] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7727.253485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7727.606058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7727.621949] Lustre: Skipped 1 previous similar message [ 7734.417758] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7744.992542] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7756.321872] LustreError: 161675:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x240000bd0:0x2:0x0]: rc = -108 [ 7756.352940] Lustre: 161675:0:(llog_cat_server.c:670:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 records in [0x240000bd0:0x2:0x0]: rc = -108 [ 7756.687329] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.12@tcp (stopping) [ 7758.326063] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7758.347304] Lustre: Skipped 2 previous similar messages [ 7762.881965] Lustre: server umount lustre-MDT0000 complete [ 7763.435459] LustreError: 164144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7763.452228] LustreError: 164144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7767.154920] LustreError: 161629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788484810 with bad export cookie 17526106635861451675 [ 7767.164761] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7767.167264] LustreError: 161629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7768.551319] LustreError: 161643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7768.552039] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7768.591334] LustreError: 161643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7773.658599] Lustre: server umount lustre-MDT0001 complete [ 7781.877149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7797.384933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7797.582987] LustreError: 165494:0:(llog.c:1655:llog_backup()) MGC192.168.204.112@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7797.589749] Lustre: 165494:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.112@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7801.775072] LustreError: 165499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7801.795718] LustreError: 165499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7818.107893] LustreError: 165501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7818.130128] LustreError: 165501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7852.905803] LustreError: 165501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7852.918038] LustreError: 165501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 7858.144338] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7858.158521] Lustre: 165494:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.112@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7899.129853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7904.248249] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7904.666674] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7914.541111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7917.536208] LustreError: 165499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7917.559809] LustreError: 165499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 46 previous similar messages [ 7927.792994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7943.283016] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7943.665698] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7951.406926] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7953.398409] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7960.537779] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7963.008220] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7989.729126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7989.735597] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7989.747674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7989.759851] Lustre: Skipped 1 previous similar message [ 7989.777657] Lustre: Skipped 9 previous similar messages [ 7994.735831] Lustre: server umount lustre-MDT0000 complete [ 7999.943270] LustreError: 161629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788485042 with bad export cookie 17526106635861458164 [ 7999.961841] LustreError: 161629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7999.962355] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8000.406613] Lustre: server umount lustre-MDT0001 complete [ 8007.181648] Lustre: server umount lustre-OST0000 complete [ 8032.574654] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 8046.607906] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 8090.652914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8091.123506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8091.153889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8091.300281] Lustre: lustre-MDT0000: new disk, initializing [ 8091.406320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8091.418162] Lustre: Skipped 1 previous similar message [ 8091.437703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8097.963570] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8110.814969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8111.075839] Lustre: 170616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8111.525305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8117.216944] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8129.171992] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8135.148988] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8143.339152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8143.566769] Lustre: lustre-OST0000: new disk, initializing [ 8143.570372] Lustre: Skipped 1 previous similar message [ 8143.576059] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8143.584477] Lustre: Skipped 2 previous similar messages [ 8143.587762] Lustre: 172585:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8149.070993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8149.090382] Lustre: Skipped 1 previous similar message [ 8149.093440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8149.187968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8150.039680] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8160.828708] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8164.827857] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8165.006348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8169.991353] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8170.264279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8175.071890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8175.078508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8175.090233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8175.090241] Lustre: Skipped 1 previous similar message [ 8175.115564] Lustre: Skipped 3 previous similar messages [ 8178.711630] Lustre: server umount lustre-OST0000 complete [ 8183.272126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8189.089952] Lustre: server umount lustre-MDT0000 complete [ 8190.432512] LustreError: 170626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8190.447147] LustreError: 170626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [ 8193.260274] LustreError: 170609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788485236 with bad export cookie 17526106635861474971 [ 8193.266124] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8193.272729] LustreError: 170609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8193.683621] Lustre: server umount lustre-MDT0001 complete [ 8205.203259] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:27:25 (1788485245) [ 8214.108942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8220.155490] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8229.872812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8234.958865] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8243.494575] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8249.784744] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8257.882547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8264.690364] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8272.383516] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8286.289274] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:28:46 (1788485326) [ 8291.311240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8291.318649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8291.343477] Lustre: Skipped 3 previous similar messages [ 8297.391981] Lustre: server umount lustre-OST0000 complete [ 8306.671369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8306.677774] Lustre: Skipped 9 previous similar messages [ 8311.700312] Lustre: server umount lustre-MDT0000 complete [ 8316.578261] LustreError: 174979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788485359 with bad export cookie 17526106635861475853 [ 8316.612269] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8317.101809] Lustre: server umount lustre-MDT0001 complete [ 8334.338623] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 8338.342372] Key type lgssc unregistered [ 8338.674905] LNet: 178731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8338.687796] LNetError: 178731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8338.712964] LNet: Removed LNI 192.168.204.112@tcp [ 8339.792254] Key type .llcrypt unregistered [ 8339.794994] Key type ._llcrypt unregistered [ 8361.035977] Key type ._llcrypt registered [ 8361.038591] Key type .llcrypt registered [ 8361.155959] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 8362.801713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8362.851505] alg: No test for adler32 (adler32-zlib) [ 8363.898311] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 8364.252110] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 8366.031179] Key type lgssc registered [ 8367.399197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8376.007527] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8376.035685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8391.711660] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8396.831309] LustreError: 179970:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.112@tcp: failed processing log, type 4: rc = -110 [ 8401.952725] LustreError: 179970:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8401.973781] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8401.985220] LustreError: MGC192.168.204.112@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8402.009360] LustreError: 179970:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8402.024458] LustreError: 179970:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 8402.038382] LustreError: 179970:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8402.125345] Lustre: server umount lustre-OST0000 complete [ 8402.127411] LustreError: 179970:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8410.243676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8411.703477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8415.845895] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8421.863524] LustreError: 180470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8421.904133] LustreError: 180470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8423.654564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8423.998312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8428.280663] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8435.539538] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8441.090573] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8448.733191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8449.087069] LustreError: 182285:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8449.100984] Lustre: 182285:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.112@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8449.263178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8455.742891] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8463.851045] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8466.912756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8466.920153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8466.935658] Lustre: Skipped 1 previous similar message [ 8466.944930] Lustre: Skipped 2 previous similar messages [ 8472.035718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8472.039546] Lustre: Skipped 1 previous similar message [ 8472.938801] Lustre: server umount lustre-MDT0000 complete [ 8477.155657] LustreError: 180469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8477.171392] LustreError: 180469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8478.479721] LustreError: 180450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788485521 with bad export cookie 8782455529004879646 [ 8478.482964] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8478.489645] LustreError: 180450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8479.101457] Lustre: server umount lustre-MDT0001 complete [ 8496.202575] Lustre: server umount lustre-OST0000 complete [ 8508.479873] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 8520.246405] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 8557.633400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8557.810474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8557.841555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8557.910927] Lustre: lustre-MDT0000: new disk, initializing [ 8557.961846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8557.976081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8562.416958] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8576.541548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8576.704857] Lustre: 185811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8576.807640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8576.826119] Lustre: Skipped 1 previous similar message [ 8577.060268] Lustre: lustre-MDT0001: new disk, initializing [ 8577.258431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8577.298779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8577.321987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8583.051476] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8592.141595] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8597.709155] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8603.992614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8604.302376] Lustre: lustre-OST0000: new disk, initializing [ 8604.307604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8604.315513] Lustre: 187781:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8604.439473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8609.925343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8609.944026] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8610.081897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8610.210460] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8619.127494] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8624.130593] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8624.353100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8629.471748] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8629.657150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8635.877863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8635.888827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8635.896868] Lustre: Skipped 1 previous similar message [ 8635.914118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8635.929842] Lustre: Skipped 2 previous similar messages [ 8637.946815] Lustre: server umount lustre-OST0000 complete [ 8644.065555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8644.078488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8644.107298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8646.112527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8651.235222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8651.244416] Lustre: Skipped 3 previous similar messages [ 8656.351489] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8656.551505] Lustre: server umount lustre-MDT0000 complete [ 8660.395932] LustreError: 185803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788485703 with bad export cookie 8782455529004880367 [ 8660.398797] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8660.402798] LustreError: 185803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8660.712181] Lustre: server umount lustre-MDT0001 complete [ 8669.664756] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 21:35:10 (1788485710) [ 8688.213780] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 8691.021286] Key type lgssc unregistered [ 8691.358228] LNet: 190726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8691.367617] LNetError: 190726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8691.401441] LNet: Removed LNI 192.168.204.112@tcp [ 8692.328254] Key type .llcrypt unregistered [ 8692.331914] Key type ._llcrypt unregistered [ 8710.260403] Key type ._llcrypt registered [ 8710.262295] Key type .llcrypt registered [ 8710.355837] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 8711.252857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8711.314676] alg: No test for adler32 (adler32-zlib) [ 8712.346875] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 8712.678413] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 8714.367194] Key type lgssc registered [ 8715.451876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8722.735922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8722.751249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8724.123492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8728.688530] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8733.675272] LustreError: 191945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8733.696986] LustreError: 191945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8736.736894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8737.133255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8741.642463] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8749.188729] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8754.653836] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8762.748986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8762.928747] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8762.935596] LustreError: 193809:0:(llog.c:1696:llog_backup()) MGC192.168.204.112@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8762.943934] Lustre: 193809:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.112@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8763.054510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8770.394109] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8778.348788] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8781.670876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8781.679479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8781.692324] Lustre: Skipped 2 previous similar messages [ 8781.708218] Lustre: Skipped 2 previous similar messages [ 8786.915732] LustreError: 193613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8786.938107] LustreError: 193613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8787.077331] Lustre: server umount lustre-MDT0000 complete [ 8790.819392] LustreError: 191927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788485833 with bad export cookie 11731072666214568458 [ 8790.829083] LustreError: 191927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8790.829436] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8791.967519] LustreError: 192674:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002656309c x1875363239062912/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8792.046352] LustreError: 191940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8792.046786] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8792.163357] Lustre: server umount lustre-MDT0001 complete [ 8798.256844] Lustre: server umount lustre-OST0000 complete [ 8809.022753] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 8820.495303] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 8861.142787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8861.328493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8861.361908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8861.439637] Lustre: lustre-MDT0000: new disk, initializing [ 8861.515763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8861.537564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8865.852884] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8877.562857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8877.763028] Lustre: 197331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8877.843508] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8877.846244] Lustre: Skipped 1 previous similar message [ 8877.854444] LustreError: 198114:0:(llog.c:1696:llog_backup()) MGC192.168.204.112@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8877.881350] LustreError: 198114:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8877.890125] Lustre: 198114:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.112@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8877.921180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8877.930565] Lustre: Skipped 1 previous similar message [ 8878.052679] Lustre: lustre-MDT0001: new disk, initializing [ 8878.138300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8878.195932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8878.212404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8884.181968] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8896.075814] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8902.656704] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8909.658689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8909.845734] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8909.850319] Lustre: Skipped 1 previous similar message [ 8909.855049] LustreError: 199279:0:(llog.c:1696:llog_backup()) MGC192.168.204.112@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8909.864197] LustreError: 199279:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8909.873836] Lustre: 199279:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.112@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8909.885938] Lustre: 199279:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8909.922671] Lustre: lustre-OST0000: new disk, initializing [ 8909.926230] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8909.932202] Lustre: 199302:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8910.036302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8916.063595] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 8919.644919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8919.655122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8919.802613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8925.659345] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8930.428373] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8930.618944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8934.615720] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8934.838388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8940.515530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8940.520044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8940.532122] LustreError: Skipped 1 previous similar message [ 8940.535126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8940.562994] Lustre: Skipped 1 previous similar message [ 8942.758802] Lustre: server umount lustre-OST0000 complete [ 8949.730023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8949.737982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8949.752883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8949.759126] Lustre: Skipped 1 previous similar message [ 8953.034653] Lustre: server umount lustre-MDT0000 complete [ 8955.876713] LustreError: 197343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8955.896984] LustreError: 197343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8957.090461] LustreError: 198126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788485999 with bad export cookie 11731072666214569172 [ 8957.096399] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8957.104817] LustreError: 198126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8957.548666] Lustre: server umount lustre-MDT0001 complete [ 8966.975898] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 21:40:07 (1788486007) [ 8984.991118] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 8996.519173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8996.929075] LustreError: 202430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8997.001350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9001.965992] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9006.566100] LustreError: 202431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9006.588795] LustreError: 202431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9012.547197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9017.951690] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9020.959923] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9028.116706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9035.198921] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9045.383253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9045.563596] Lustre: lustre-OST0001: new disk, initializing [ 9045.571311] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9045.584758] Lustre: 204739:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9045.718163] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9045.723992] Lustre: Skipped 2 previous similar messages [ 9051.739887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9051.756846] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9051.855233] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9052.181290] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9062.842395] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9071.469784] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9073.661436] Lustre: DEBUG MARKER: ADD OST3 [ 9079.057438] loop: module loaded [ 9079.156658] loop0: detected capacity change from 0 to 204800000 [ 9079.454705] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9079.473068] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9079.492469] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9085.587412] loop0: detected capacity change from 0 to 204800000 [ 9086.711754] Lustre: DEBUG MARKER: STOP OST3 [ 9092.643219] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9092.827766] Lustre: lustre-OST0003: new disk, initializing [ 9092.833358] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 9092.841391] Lustre: 206731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9094.678518] Lustre: *** cfs_fail_loc=2109, val=2*** [ 9094.862094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 9094.876374] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 9094.925552] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 9096.735545] LustreError: 206903:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 9097.086284] Lustre: Failing over lustre-OST0003 [ 9097.196975] Lustre: server umount lustre-OST0003 complete [ 9100.163221] LustreError: 204747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9100.183147] LustreError: 204747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9100.277291] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9100.282236] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9100.292704] Lustre: Skipped 1 previous similar message [ 9101.131315] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9109.593470] Lustre: DEBUG MARKER: START OST3 again [ 9116.714831] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9117.177862] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 9117.189725] Lustre: Skipped 1 previous similar message [ 9117.207847] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 9118.897240] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9119.227116] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9119.227505] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9123.907708] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9145.399209] Lustre: Failing over lustre-OST0003 [ 9145.573693] Lustre: server umount lustre-OST0003 complete [ 9147.894137] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9147.916028] Lustre: Skipped 1 previous similar message [ 9147.932993] LustreError: 203916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9147.956902] LustreError: 203916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 9166.305042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9166.313240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9166.329127] Lustre: Skipped 1 previous similar message [ 9166.337474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9166.345126] Lustre: Skipped 2 previous similar messages [ 9170.402371] Lustre: server umount lustre-MDT0000 complete [ 9173.414219] LustreError: 204738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788486216 with bad export cookie 11731072666214570061 [ 9173.426284] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9173.431567] LustreError: 204738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9173.483170] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9173.487330] Lustre: Skipped 5 previous similar messages [ 9179.845879] Lustre: server umount lustre-MDT0001 complete [ 9183.410385] Lustre: server umount lustre-OST0000 complete [ 9187.067230] Lustre: server umount lustre-OST0001 complete [ 9195.124807] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 9205.932178] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 9243.434879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9243.703519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9243.733789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9243.794285] Lustre: lustre-MDT0000: new disk, initializing [ 9243.987417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9249.804691] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9259.860139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9259.964639] Lustre: 212199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9260.005854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9260.009445] Lustre: Skipped 1 previous similar message [ 9260.064242] Lustre: lustre-MDT0001: new disk, initializing [ 9260.139365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9260.144456] Lustre: Skipped 1 previous similar message [ 9260.168797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9260.180023] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9264.682315] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9273.544252] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9279.030206] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9286.153132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9286.444407] Lustre: 214171:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9287.897787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9288.015345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9292.379049] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9302.827623] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9308.123684] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9308.345905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9313.088366] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9313.377956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9318.881292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9318.890244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9318.902979] Lustre: Skipped 5 previous similar messages [ 9318.909604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9318.925266] Lustre: Skipped 3 previous similar messages [ 9321.498772] Lustre: server umount lustre-OST0000 complete [ 9338.337467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9338.358968] Lustre: Skipped 6 previous similar messages [ 9339.359155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9339.546736] Lustre: server umount lustre-MDT0000 complete [ 9343.459558] LustreError: 212212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9343.490409] LustreError: 212212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 9344.123438] LustreError: 214170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788486386 with bad export cookie 11731072666214572357 [ 9344.154251] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9344.569149] Lustre: server umount lustre-MDT0001 complete [ 9349.049723] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:46:29 (1788486389) [ 9366.239586] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 9376.243780] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 9411.189239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9411.593260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9411.646169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9411.661289] Lustre: Skipped 1 previous similar message [ 9411.799325] Lustre: lustre-MDT0000: new disk, initializing [ 9411.809880] Lustre: Skipped 1 previous similar message [ 9411.923836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9411.928469] Lustre: Skipped 1 previous similar message [ 9416.582646] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9427.871592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9427.973926] Lustre: 218783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9428.250621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9432.539354] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9442.022246] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9446.812644] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9454.006124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9454.278238] Lustre: 220750:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9455.639513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9460.631576] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9471.602236] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9476.908901] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9477.094502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9481.885341] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9482.050708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9486.307473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9486.314911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9486.327175] Lustre: Skipped 3 previous similar messages [ 9486.332530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9486.336265] LustreError: Skipped 3 previous similar messages [ 9486.345507] Lustre: Skipped 1 previous similar message [ 9498.079137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9498.423459] Lustre: server umount lustre-OST0000 complete [ 9505.251212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9508.625521] Lustre: server umount lustre-MDT0000 complete [ 9511.908097] LustreError: 218794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9511.924605] LustreError: 218794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9511.974892] LustreError: 219585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788486554 with bad export cookie 11731072666214573239 [ 9511.978874] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9512.005353] LustreError: 219585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9512.426759] Lustre: server umount lustre-MDT0001 complete [ 9522.087432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9522.864397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9522.876149] Lustre: Skipped 4 previous similar messages [ 9528.355286] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9539.787237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9547.074854] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9558.040887] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9566.838620] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9574.140316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9581.355862] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9589.634775] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9605.600346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9605.608479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9605.620034] Lustre: Skipped 7 previous similar messages [ 9611.405430] Lustre: server umount lustre-OST0000 complete [ 9616.864169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9616.877456] Lustre: Skipped 5 previous similar messages [ 9629.154395] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9629.326990] Lustre: server umount lustre-MDT0000 complete [ 9632.641146] LustreError: 222950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788486675 with bad export cookie 11731072666214574121 [ 9632.653755] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9632.656168] LustreError: 222950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9633.096199] Lustre: server umount lustre-MDT0001 complete [ 9729.713497] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 9733.150765] Key type lgssc unregistered [ 9733.509496] LNet: 226651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9733.530230] LNetError: 226651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9733.547943] LNet: Removed LNI 192.168.204.112@tcp [ 9734.618950] Key type .llcrypt unregistered [ 9734.621620] Key type ._llcrypt unregistered [ 9760.427285] Key type ._llcrypt registered [ 9760.433836] Key type .llcrypt registered [ 9760.533737] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 9775.854610] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 9776.804791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9776.938754] alg: No test for adler32 (adler32-zlib) [ 9778.021273] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 9778.230239] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 9779.896864] Key type lgssc registered [ 9781.044033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9814.537696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9814.555169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9815.814471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9815.849524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9815.950721] Lustre: lustre-MDT0000: new disk, initializing [ 9816.049952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9816.069894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9820.664208] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9832.166692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9832.289176] Lustre: 229998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9832.334462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9832.338906] Lustre: Skipped 1 previous similar message [ 9832.398278] Lustre: lustre-MDT0001: new disk, initializing [ 9832.485974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9832.524991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9832.549316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9837.876207] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9848.597817] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9854.580938] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9861.841593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9862.141971] Lustre: lustre-OST0000: new disk, initializing [ 9862.145068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9862.151374] Lustre: 231965:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9862.279826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9867.163450] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9871.992353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9872.004433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9872.136801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9876.185955] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9881.447087] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9881.678748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9885.979615] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9886.170794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9892.834425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9892.834751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9892.844918] LustreError: Skipped 1 previous similar message [ 9892.848166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9892.857037] Lustre: Skipped 1 previous similar message [ 9892.875455] Lustre: Skipped 1 previous similar message [ 9894.176588] Lustre: server umount lustre-OST0000 complete [ 9898.979803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9898.992805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9899.001973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9903.073286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9903.074441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9903.092655] Lustre: Skipped 1 previous similar message [ 9908.194639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9908.201895] Lustre: Skipped 1 previous similar message [ 9912.288788] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9912.291491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9912.317873] Lustre: Skipped 1 previous similar message [ 9912.512099] Lustre: server umount lustre-MDT0000 complete [ 9916.452056] LustreError: 231964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788486959 with bad export cookie 3616324402273088818 [ 9916.452792] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9916.461638] LustreError: 231964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9917.118899] Lustre: server umount lustre-MDT0001 complete [ 9938.477987] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 9949.881603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9950.449865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9955.514628] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9955.809546] LustreError: 234905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9955.825081] LustreError: 234905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9960.930964] LustreError: 234904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9964.990158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9965.335972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9971.223453] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 9974.992236] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9982.649670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9983.264066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9990.609939] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10002.283794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10002.522359] Lustre: lustre-OST0001: new disk, initializing [10002.526831] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10002.533207] Lustre: 237212:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10002.628093] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10010.212614] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10010.793451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10010.806519] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10010.915083] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10022.176443] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10035.165153] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10049.939530] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:58:10 (1788487090) [10052.576788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10052.588184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10052.616686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10057.188321] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10057.212916] Lustre: Skipped 2 previous similar messages [10058.739912] Lustre: server umount lustre-MDT0000 complete [10062.306868] LustreError: 234901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10062.336242] LustreError: 234901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10062.687236] LustreError: 234886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788487105 with bad export cookie 3616324402273089693 [10062.692306] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10062.710891] LustreError: 234886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10063.225474] Lustre: server umount lustre-MDT0001 complete [10077.886890] Lustre: server umount lustre-OST0000 complete [10092.569550] Lustre: server umount lustre-OST0001 complete [10099.593901] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [10110.213457] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [10158.664828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10158.930983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10159.017267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10159.088737] Lustre: lustre-MDT0000: new disk, initializing [10159.177168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10159.198541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10164.146732] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10174.684562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10174.858618] Lustre: 241163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10174.921893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10174.925270] Lustre: Skipped 1 previous similar message [10175.049278] Lustre: lustre-MDT0001: new disk, initializing [10175.184712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10175.194122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10180.362686] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10190.281744] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10196.648102] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10203.700330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10203.987917] Lustre: lustre-OST0000: new disk, initializing [10203.992810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10204.002277] Lustre: 243133:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10206.051962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10206.075858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10206.139489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10210.703449] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10222.629710] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10228.125856] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10228.393285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10233.621185] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10233.839906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10236.896252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10236.899952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10236.917744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10236.925090] Lustre: Skipped 1 previous similar message [10236.939420] Lustre: Skipped 5 previous similar messages [10242.020601] Lustre: server umount lustre-OST0000 complete [10247.140198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10252.403983] Lustre: server umount lustre-MDT0000 complete [10256.358447] LustreError: 241175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.374030] LustreError: 241175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10256.394403] LustreError: 243132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788487299 with bad export cookie 3616324402273091835 [10256.401573] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10256.409584] LustreError: 243132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10256.948911] Lustre: server umount lustre-MDT0001 complete [10265.970045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10266.332622] LustreError: 245361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10266.441062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10266.449144] Lustre: Skipped 2 previous similar messages [10272.117635] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10276.832880] LustreError: 245361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10276.877422] LustreError: 245361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10281.593483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10286.897215] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10294.934306] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10300.196584] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10306.983662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10313.576805] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10321.226551] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10470.378184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10470.384329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10470.396475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10470.398222] Lustre: Skipped 3 previous similar messages [10470.410250] Lustre: Skipped 5 previous similar messages [10472.563174] Lustre: server umount lustre-OST0000 complete [10491.359207] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10491.519781] Lustre: server umount lustre-MDT0000 complete [10495.066148] LustreError: 245344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788487537 with bad export cookie 3616324402273092717 [10495.074088] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10495.082549] LustreError: 245344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10495.339829] Lustre: server umount lustre-MDT0001 complete [10589.779757] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [10592.557813] Key type lgssc unregistered [10592.820551] LNet: 248967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10592.835551] LNetError: 248967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10592.857751] LNet: Removed LNI 192.168.204.112@tcp [10593.772232] Key type .llcrypt unregistered [10593.777401] Key type ._llcrypt unregistered [10621.003811] Key type ._llcrypt registered [10621.006844] Key type .llcrypt registered [10621.117789] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [10642.573246] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [10643.878239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10643.978283] alg: No test for adler32 (adler32-zlib) [10645.189253] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [10645.452634] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [10647.095366] Key type lgssc registered [10648.553856] Lustre: Echo OBD driver; http://www.lustre.org/ [10703.135088] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [10715.499043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10715.538087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10716.799407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10716.836885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10716.959834] Lustre: lustre-MDT0000: new disk, initializing [10717.074603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10717.087508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10722.310389] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10734.938165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10735.166721] Lustre: 253051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10735.205326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10735.210059] Lustre: Skipped 1 previous similar message [10735.278191] Lustre: lustre-MDT0001: new disk, initializing [10735.368656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10735.428641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10735.449447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10741.416364] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10748.527969] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10758.657340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10758.856499] Lustre: lustre-OST0000: new disk, initializing [10758.862289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10758.869570] Lustre: 254688:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10758.998737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10764.962139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10764.974716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10765.136297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10766.199401] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10781.373442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10781.621511] Lustre: lustre-OST0001: new disk, initializing [10781.624131] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10781.631447] Lustre: 255560:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10781.718588] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10787.546503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10787.561608] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10787.707385] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10789.751460] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10802.675305] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10809.109390] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10823.649932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10823.660663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10823.664181] Lustre: Skipped 2 previous similar messages [10828.776301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10828.783636] Lustre: Skipped 4 previous similar messages [10829.429098] Lustre: server umount lustre-MDT0000 complete [10833.719065] LustreError: 253044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788487876 with bad export cookie 11718782841537347556 [10833.719939] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10833.723303] LustreError: 253044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10833.889893] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10833.908409] Lustre: Skipped 4 previous similar messages [10834.003159] Lustre: server umount lustre-MDT0001 complete [10844.278813] Lustre: server umount lustre-OST0000 complete [10848.653187] Lustre: server umount lustre-OST0001 complete [10864.881543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10876.112157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10885.905888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10897.157318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10917.630245] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [10928.399941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10928.414995] Lustre: lustre-MDT0000: reset Object Index mappings [10928.754305] LustreError: 260565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10928.761937] LustreError: 260565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10928.805696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10934.256430] LustreError: 260566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10934.412955] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10938.345510] LustreError: 260565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10943.376951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10943.410200] Lustre: lustre-MDT0001: reset Object Index mappings [10943.462205] LustreError: 260566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10943.835346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10948.268639] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10950.916637] Lustre: 261708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10957.522142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10963.828481] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10969.073851] LustreError: 262061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10973.172263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10973.834467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10974.189299] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10974.193250] Lustre: Skipped 1 previous similar message [10979.310616] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10981.284133] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [10989.658967] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10993.916034] Lustre: 263579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10999.779964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10999.792386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10999.813142] Lustre: Skipped 2 previous similar messages [10999.827102] Lustre: Skipped 3 previous similar messages [11004.898477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11004.910807] Lustre: Skipped 1 previous similar message [11005.865267] Lustre: server umount lustre-MDT0000 complete [11010.022331] LustreError: 260562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11010.046496] LustreError: 260562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11010.212688] LustreError: 263582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788488053 with bad export cookie 11718782841537354346 [11010.221174] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11010.227972] LustreError: 263582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11010.684478] Lustre: server umount lustre-MDT0001 complete [11025.565033] Lustre: server umount lustre-OST0000 complete [11040.131146] Lustre: server umount lustre-OST0001 complete [11061.819807] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 22:15:02 (1788488102) [11081.315947] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [11093.376654] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [11144.992336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11145.458824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11145.520267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11145.664826] Lustre: lustre-MDT0000: new disk, initializing [11145.846169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11145.886636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11151.237121] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11162.283415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11162.495414] Lustre: 267843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11162.563344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11162.566979] Lustre: Skipped 1 previous similar message [11162.632834] Lustre: lustre-MDT0001: new disk, initializing [11162.851744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11162.858963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11167.616819] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11178.205881] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11184.088059] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11191.506064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11191.797339] Lustre: lustre-OST0000: new disk, initializing [11191.802452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11191.809713] Lustre: 269809:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11193.069067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11193.080624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11193.208353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11198.170653] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11208.138466] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11213.905599] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11214.136481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11219.949745] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11220.211166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11224.045483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11224.046626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11224.054203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11224.077783] Lustre: Skipped 2 previous similar messages [11228.660298] Lustre: server umount lustre-OST0000 complete [11234.278544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11234.301320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11234.304251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11234.304260] Lustre: Skipped 1 previous similar message [11234.317304] Lustre: Skipped 1 previous similar message [11234.331592] LustreError: Skipped 1 previous similar message [11239.274567] Lustre: server umount lustre-MDT0000 complete [11239.396442] LustreError: 267855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11239.430775] LustreError: 267855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11243.709122] LustreError: 267835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788488286 with bad export cookie 11718782841537356439 [11243.713076] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11243.740736] LustreError: 267835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11244.256347] Lustre: server umount lustre-MDT0001 complete [11269.113906] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [11281.249408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11281.697442] LustreError: 272745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11281.799410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11281.805563] Lustre: Skipped 2 previous similar messages [11288.560434] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11300.089703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11306.049287] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11309.251730] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11317.339748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11324.681697] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11334.104444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11334.351669] Lustre: lustre-OST0001: new disk, initializing [11334.356520] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11334.361685] Lustre: 275059:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11335.540811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11335.560184] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11335.678087] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11341.840598] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11352.374746] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11356.705495] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11362.695663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11362.777787] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11376.610433] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11376.616943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11376.627431] Lustre: Skipped 2 previous similar messages [11376.635796] Lustre: Skipped 4 previous similar messages [11380.613715] Lustre: server umount lustre-MDT0000 complete [11381.731993] LustreError: 274670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11381.746742] LustreError: 274670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11385.091567] LustreError: 272727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788488427 with bad export cookie 11718782841537357321 [11385.093331] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11385.106554] LustreError: 272727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11385.624939] Lustre: server umount lustre-MDT0001 complete [11400.909878] Lustre: server umount lustre-OST0000 complete [11402.719792] Lustre: 250541:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788488429/real 1788488429] req@ffff9c63bc2d4e00 x1875365266183296/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788488445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11402.770469] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11402.795943] Lustre: Skipped 1 previous similar message [11405.884868] Lustre: server umount lustre-OST0001 complete [11427.821032] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [11440.763445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11441.232202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11441.239898] Lustre: Skipped 3 previous similar messages [11446.710308] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11457.041305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11462.560753] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11465.888141] Lustre: 279547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11473.432849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11480.986497] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11487.192065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11491.733579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11497.494685] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11500.371943] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11508.631379] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11512.750267] Lustre: 281423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11518.302256] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11518.368403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11533.528752] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 22:22:53 (1788488573) [11549.665712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11549.672643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11549.689297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11555.682811] Lustre: server umount lustre-MDT0000 complete [11558.783299] LustreError: 278388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788488601 with bad export cookie 11718782841537364223 [11558.790031] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11558.798041] LustreError: 278388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11558.881159] LustreError: 281730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11558.900229] LustreError: 281730:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [11573.215121] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11573.556077] Lustre: server umount lustre-MDT0001 complete [11584.210214] Lustre: server umount lustre-OST0000 complete [11588.560539] Lustre: server umount lustre-OST0001 complete [11597.155758] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [11607.902898] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [11661.291944] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [11671.572471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11671.839037] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11671.895742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11672.038399] Lustre: lustre-MDT0000: new disk, initializing [11672.202185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11676.982831] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11688.731808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11688.913725] Lustre: 285955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11689.169431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11693.999738] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11708.144338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11708.561370] Lustre: 287590:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11708.696366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11708.703967] Lustre: Skipped 5 previous similar messages [11709.972317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11716.707599] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11728.668441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11728.914156] Lustre: 288462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11730.166435] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11730.196151] Lustre: Skipped 1 previous similar message [11730.319376] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11736.003636] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11747.332796] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11751.522627] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11751.530218] Lustre: Skipped 1 previous similar message [11769.084584] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 22:26:49 (1788488809) [11774.040155] Lustre: server umount lustre-MDT0000 complete [11774.433593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11774.445835] Lustre: Skipped 8 previous similar messages [11778.176903] LustreError: 286751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788488821 with bad export cookie 11718782841537371713 [11778.184704] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11778.191903] LustreError: 286751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11779.085713] Lustre: server umount lustre-MDT0001 complete [11793.381914] Lustre: server umount lustre-OST0000 complete [11795.743122] Lustre: 250542:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788488822/real 1788488822] req@ffff9c6293ad5180 x1875365266342528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788488838 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11798.591243] Lustre: server umount lustre-OST0001 complete [11810.801546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11816.933123] LustreError: 290711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11816.953954] LustreError: 290711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11817.244769] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11827.551242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11833.102455] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11842.667757] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11849.799304] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11857.878460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11866.576782] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11876.959416] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11901.316271] loop1: detected capacity change from 0 to 204800000 [11901.629329] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11901.651814] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11901.666945] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11907.703484] loop1: detected capacity change from 0 to 204800000 [11914.661833] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11914.819551] LustreError: 290701:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs26257: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11914.841265] LustreError: 290701:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs26257 [11914.846809] LustreError: 290701:0:(mgs_handler.c:517:mgs_target_reg()) fs26257-MDT0001: Error checking target index: rc = -13 [11914.855384] LustreError: fs26257-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11914.875631] LustreError: 294430:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11914.881832] LustreError: 294430:0:(tgt_mount.c:2132:server_put_super()) no obd fs26257-MDT0001 [11914.890896] LustreError: 294430:0:(tgt_mount.c:116:server_deregister_mount()) fs26257-MDT0001 not registered [11914.927255] Lustre: server umount fs26257-MDT0001 complete [11914.935392] LustreError: 294430:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11925.464076] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11925.572263] Lustre: Setting parameter fs26257-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs26257-MDT0001 [11925.633398] Lustre: srv-fs26257-MDT0001: No data found on store. Initialize space. [11925.639027] Lustre: Skipped 4 previous similar messages [11925.712539] Lustre: fs26257-MDT0001: new disk, initializing [11925.716038] Lustre: Skipped 3 previous similar messages [11925.916136] Lustre: srv-fs26257-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11925.931282] Lustre: 294896:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26257-MDT0001-mdtlov: can't create new llog: rc = -115 [11925.944709] LustreError: 294896:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26257-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11931.572280] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11939.351263] Lustre: server umount fs26257-MDT0001 complete [11953.650205] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11954.227856] loop1: detected capacity change from 0 to 204800000 [11958.772213] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11961.028488] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11961.439693] Lustre: 296467:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26257-MDT0001-mdtlov: can't create new llog: rc = -115 [11961.452536] LustreError: 296467:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26257-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11967.423566] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [11973.540735] Lustre: server umount fs26257-MDT0001 complete [11985.296240] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.12@tcp (stopping) [11985.317589] Lustre: Skipped 10 previous similar messages [11986.912447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11986.921965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11986.948928] Lustre: Skipped 1 previous similar message [11991.663761] Lustre: server umount lustre-MDT0000 complete [12001.852115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12002.024427] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12007.427100] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12007.434443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12016.541888] loop1: detected capacity change from 0 to 204800000 [12023.416929] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12024.130427] Lustre: 298915:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26257-MDT0001-mdtlov: can't create new llog: rc = -115 [12024.140059] LustreError: 298915:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26257-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12029.891755] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12036.122286] Lustre: server umount fs26257-MDT0001 complete [12067.302053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12071.124953] Lustre: server umount lustre-OST0000 complete [12081.608267] Lustre: server umount lustre-MDT0000 complete [12086.129889] LustreError: 290693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788489129 with bad export cookie 11718782841537374394 [12086.134074] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12086.143129] LustreError: 290693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12086.657326] Lustre: server umount lustre-MDT0001 complete [12100.091471] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [12104.327749] Key type lgssc unregistered [12104.854762] LNet: 301258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12104.874629] LNetError: 301258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12104.899231] LNet: Removed LNI 192.168.204.112@tcp [12105.853203] Key type .llcrypt unregistered [12105.855622] Key type ._llcrypt unregistered [12130.280118] Key type ._llcrypt registered [12130.283891] Key type .llcrypt registered [12130.438554] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [12132.158665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12132.173267] alg: No test for adler32 (adler32-zlib) [12133.390416] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [12133.773072] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [12135.483885] Key type lgssc registered [12137.391280] Lustre: Echo OBD driver; http://www.lustre.org/ [12148.632075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12148.668546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12150.326525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12154.706911] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12160.482037] LustreError: 302532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12160.499472] LustreError: 302532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12165.106725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12165.839194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12170.359125] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12179.863366] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12186.571265] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12193.976153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12194.302816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12195.316569] LustreError: 304360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12200.424036] LustreError: 304362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12200.731556] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12205.538428] LustreError: 304360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12209.269223] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12210.661473] LustreError: 304362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12210.695910] LustreError: 304362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12220.914535] LustreError: 304361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12220.943374] LustreError: 304361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12228.436635] loop1: detected capacity change from 0 to 204800000 [12228.623505] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12228.658363] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12228.671283] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12234.098690] loop1: detected capacity change from 0 to 204800000 [12237.168239] LustreError: 304360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12237.181782] LustreError: 304360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [12239.580562] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12239.776303] Lustre: fs25378-OST0000: new disk, initializing [12239.787303] Lustre: srv-fs25378-OST0000: No data found on store. Initialize space. [12239.797283] Lustre: 306266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12239.889845] Lustre: fs25378-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12246.787825] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12260.035446] Lustre: server umount fs25378-OST0000 complete [12272.125912] LustreError: 304362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12272.175137] LustreError: 304362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [12276.347175] loop1: detected capacity change from 0 to 204800000 [12277.224515] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12282.766885] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12282.990265] Lustre: fs25378-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12291.242212] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12292.590107] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12301.262638] loop2: detected capacity change from 0 to 204800000 [12301.471656] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12301.481181] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12301.490152] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12307.911039] loop2: detected capacity change from 0 to 204800000 [12315.665798] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12315.814521] LustreError: fs25378-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12315.828079] LustreError: fs25378-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12315.845576] LustreError: 309310:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [12315.855828] LustreError: 309310:0:(tgt_mount.c:2132:server_put_super()) no obd fs25378-OST0001 [12315.862918] LustreError: 309310:0:(tgt_mount.c:116:server_deregister_mount()) fs25378-OST0001 not registered [12315.924727] Lustre: server umount fs25378-OST0001 complete [12315.935213] LustreError: 309310:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12329.873557] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12330.006082] Lustre: fs25378-OST0001: new disk, initializing [12330.011668] Lustre: srv-fs25378-OST0001: No data found on store. Initialize space. [12330.016608] Lustre: 309861:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12330.082836] Lustre: fs25378-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12337.461535] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12338.666071] LustreError: 304360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12338.680852] LustreError: 304360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [12348.206142] Lustre: server umount fs25378-OST0000 complete [12360.137743] Lustre: server umount fs25378-OST0001 complete [12375.078768] loop1: detected capacity change from 0 to 204800000 [12381.332935] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12381.534360] Lustre: fs25378-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12390.164976] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12399.957510] loop2: detected capacity change from 0 to 204800000 [12407.621637] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12414.598204] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12424.610429] Lustre: server umount fs25378-OST0000 complete [12430.850461] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12435.046625] Lustre: server umount fs25378-OST0001 complete [12446.196218] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12451.307381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12451.309027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12451.315563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12451.336412] Lustre: Skipped 1 previous similar message [12451.354542] Lustre: Skipped 1 previous similar message [12456.425265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12456.442532] Lustre: Skipped 1 previous similar message [12456.626976] Lustre: server umount lustre-OST0000 complete [12461.544033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12461.551237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12461.561348] Lustre: Skipped 1 previous similar message [12461.569895] Lustre: Skipped 1 previous similar message [12466.658789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12466.671849] Lustre: Skipped 1 previous similar message [12467.565785] Lustre: server umount lustre-MDT0000 complete [12471.709329] LustreError: 302510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788489514 with bad export cookie 7894471034450378069 [12471.713650] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12471.725638] LustreError: 302510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12472.255285] Lustre: server umount lustre-MDT0001 complete [12484.986823] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [12488.301216] Key type lgssc unregistered [12488.580993] LNet: 315788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12488.588900] LNetError: 315788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12489.638236] LNet: Removed LNI 192.168.204.112@tcp [12490.689158] Key type .llcrypt unregistered [12490.692790] Key type ._llcrypt unregistered [12526.433114] Key type ._llcrypt registered [12526.438591] Key type .llcrypt registered [12526.555429] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [12547.335797] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [12549.042553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12549.070662] alg: No test for adler32 (adler32-zlib) [12550.213380] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [12550.503264] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [12552.215169] Key type lgssc registered [12553.719311] Lustre: Echo OBD driver; http://www.lustre.org/ [12595.412729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12595.440182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12596.705380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12596.728257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12596.824346] Lustre: lustre-MDT0000: new disk, initializing [12596.891805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12596.909684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12602.554415] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12614.457364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12614.595180] Lustre: 319644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12614.647700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12614.652604] Lustre: Skipped 1 previous similar message [12614.714404] Lustre: lustre-MDT0001: new disk, initializing [12614.800818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12614.852859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12614.872711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12620.408377] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12632.261478] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12638.720403] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12645.693359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12646.067915] Lustre: lustre-OST0000: new disk, initializing [12646.071910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12646.078543] Lustre: 321612:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12646.229565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12653.698718] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12655.688104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12655.705584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12655.912921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12665.792801] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12668.637555] Lustre: Mounted lustre-client - version 2.17.58_2_gbb63b80 [12669.652606] Lustre: Unmounted lustre-client [12681.700108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12681.704261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12681.713731] LustreError: Skipped 1 previous similar message [12681.714597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12681.748616] Lustre: Skipped 1 previous similar message [12684.270137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12684.274965] Lustre: Skipped 1 previous similar message [12685.374473] Lustre: server umount lustre-OST0000 complete [12689.377418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12689.383605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12689.393909] Lustre: Skipped 1 previous similar message [12694.499306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12694.516871] Lustre: Skipped 2 previous similar messages [12695.661414] Lustre: server umount lustre-MDT0000 complete [12700.355162] LustreError: 319637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788489743 with bad export cookie 16252905356534922041 [12700.368625] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12700.370574] LustreError: 319637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12700.932475] Lustre: server umount lustre-MDT0001 complete [12714.437519] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [12717.344602] Key type lgssc unregistered [12719.456203] LNet: 1 local NIs in recovery (showing 1): 192.168.204.112@tcp [12719.657069] LNet: 323778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12719.664420] LNetError: 323778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12797.543542] LNet: Removed LNI 192.168.204.112@tcp [12798.585599] Key type .llcrypt unregistered [12798.593775] Key type ._llcrypt unregistered [12819.672156] Key type ._llcrypt registered [12819.673905] Key type .llcrypt registered [12819.801401] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [12839.173389] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [12840.408210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12840.488074] alg: No test for adler32 (adler32-zlib) [12841.630754] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [12841.882521] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [12843.639220] Key type lgssc registered [12844.767087] Lustre: Echo OBD driver; http://www.lustre.org/ [12882.056733] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 22:45:22 (1788489922) [12889.906110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12889.937335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12891.190729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12891.218719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12891.321179] Lustre: lustre-MDT0000: new disk, initializing [12891.443190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12891.474309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12897.353948] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12910.324561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12910.514882] Lustre: 326995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12910.635733] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12910.642475] Lustre: Skipped 1 previous similar message [12910.755748] Lustre: lustre-MDT0001: new disk, initializing [12910.888780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12910.962831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12910.986465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12915.303366] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12927.591912] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12936.179666] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12946.609400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12947.162242] Lustre: lustre-OST0000: new disk, initializing [12947.178566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12947.190510] Lustre: 328964:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12947.328347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12955.289549] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [12955.819862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12955.837253] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12956.000749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12965.890417] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12986.876111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12986.879047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12986.887748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12986.895655] Lustre: Skipped 1 previous similar message [12986.909639] LustreError: Skipped 1 previous similar message [12990.060627] Lustre: server umount lustre-OST0000 complete [12997.099775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12997.106369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12997.115747] Lustre: Skipped 1 previous similar message [12997.132583] Lustre: Skipped 2 previous similar messages [13001.129861] Lustre: server umount lustre-MDT0000 complete [13006.262208] LustreError: 326988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788490049 with bad export cookie 6898790798883332942 [13006.267089] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13006.270384] LustreError: 326988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13006.945888] Lustre: server umount lustre-MDT0001 complete [13021.292874] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [13024.728414] Key type lgssc unregistered [13025.111587] LNet: 330875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13025.124504] LNetError: 330875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13025.152171] LNet: Removed LNI 192.168.204.112@tcp [13025.973130] Key type .llcrypt unregistered [13025.975474] Key type ._llcrypt unregistered [13053.269335] Key type ._llcrypt registered [13053.271719] Key type .llcrypt registered [13053.382522] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [13054.828969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13054.907305] alg: No test for adler32 (adler32-zlib) [13055.971842] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [13056.261957] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [13057.975572] Key type lgssc registered [13059.368636] Lustre: Echo OBD driver; http://www.lustre.org/ [13069.154811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13069.181862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13070.749600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13075.869721] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [13081.062067] LustreError: 332146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13081.086122] LustreError: 332146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13086.180182] LustreError: 332145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13086.325733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13086.897452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13091.568811] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [13099.260671] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13104.645636] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13111.651650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13111.724786] LustreError: 333966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13111.980901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13117.411065] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [13124.754507] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13136.023836] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13137.514933] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13142.928191] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.12@tcp (stopping) [13143.523539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13143.535983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13143.557380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13147.121593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13147.125663] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13147.166112] Lustre: Skipped 1 previous similar message [13148.859665] Lustre: server umount lustre-MDT0000 complete [13152.225223] LustreError: 332145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13152.279678] LustreError: 332145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13153.606980] LustreError: 332127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788490196 with bad export cookie 8070071034244420077 [13153.620256] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13157.345485] LustreError: 332140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13157.347096] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13157.380084] LustreError: 332140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13157.399941] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13157.402716] Lustre: Skipped 3 previous similar messages [13160.085867] Lustre: server umount lustre-MDT0001 complete [13164.835986] Lustre: server umount lustre-OST0000 complete [13173.947288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13178.969977] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [13182.235305] Lustre: 336328:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.112@tcp [13184.964530] Lustre: server umount MGS complete [13197.132692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13197.713356] LustreError: 336900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13197.724140] LustreError: 336900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13197.770349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13203.580589] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [13212.323489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13212.637434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13216.515553] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [13226.519336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13227.301818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13235.192293] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [13245.589243] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13247.440923] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13253.662997] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13255.436233] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13261.805830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [13261.816665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13261.852040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13261.864790] Lustre: Skipped 2 previous similar messages [13262.512076] Lustre: server umount lustre-OST0000 complete [13268.460958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13268.476562] Lustre: Skipped 1 previous similar message [13273.104472] Lustre: server umount lustre-MDT0000 complete [13273.573701] LustreError: 336901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13273.591469] LustreError: 336901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [13277.371229] LustreError: 336881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788490320 with bad export cookie 8070071034244420917 [13277.377332] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13277.386638] LustreError: 336881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13288.049659] Lustre: server umount lustre-MDT0001 complete [13299.436813] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [13305.101714] Key type lgssc unregistered [13305.835489] LNet: 340337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13305.858586] LNetError: 340337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13305.915487] LNet: Removed LNI 192.168.204.112@tcp [13307.529647] Key type .llcrypt unregistered [13307.533493] Key type ._llcrypt unregistered [13336.130226] Key type ._llcrypt registered [13336.133737] Key type .llcrypt registered [13336.292652] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [13357.583569] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [13359.010177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13359.039123] alg: No test for adler32 (adler32-zlib) [13360.286288] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [13360.649221] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [13362.431552] Key type lgssc registered [13364.216635] Lustre: Echo OBD driver; http://www.lustre.org/ [13405.915086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13405.947721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13407.280144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13407.320879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13407.444948] Lustre: lustre-MDT0000: new disk, initializing [13407.603632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13407.636577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13413.125240] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [13425.952809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13426.116526] Lustre: 343501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13426.238552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13426.247943] Lustre: Skipped 1 previous similar message [13426.387956] Lustre: lustre-MDT0001: new disk, initializing [13426.567870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13426.627232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13426.653245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13432.134426] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [13444.264109] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13450.769944] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13459.084672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13459.295664] Lustre: lustre-OST0000: new disk, initializing [13459.298639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13459.303341] Lustre: 345467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13459.417773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13466.974616] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [13468.237926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13468.254323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13468.376459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13477.900955] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13483.108188] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13483.329300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13488.257359] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13488.454646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13494.240512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13494.249776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13494.252937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13494.265034] Lustre: Skipped 1 previous similar message [13494.277864] Lustre: Skipped 1 previous similar message [13496.584365] Lustre: server umount lustre-OST0000 complete [13503.458286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13503.470534] LustreError: Skipped 1 previous similar message [13503.473978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13503.501714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13504.487663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13504.512124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13504.523674] Lustre: Skipped 1 previous similar message [13509.600193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13509.608611] Lustre: Skipped 1 previous similar message [13514.724862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13514.737263] Lustre: Skipped 1 previous similar message [13515.232108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13515.450720] Lustre: server umount lustre-MDT0000 complete [13518.989767] LustreError: 343491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788490561 with bad export cookie 11705534816542795260 [13518.993773] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13519.019140] LustreError: 343491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13519.388218] Lustre: server umount lustre-MDT0001 complete [13531.512980] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 22:56:11 (1788490571) [13549.859788] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [13562.777512] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [13607.146394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13607.480954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13607.514730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13607.624494] Lustre: lustre-MDT0000: new disk, initializing [13607.798821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13607.826401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13613.766310] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [13625.609762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13625.746598] Lustre: 350231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13625.793426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13625.797243] Lustre: Skipped 1 previous similar message [13625.858687] Lustre: lustre-MDT0001: new disk, initializing [13625.939737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13626.004898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13626.018500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13631.023482] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [13641.762486] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13648.455745] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13656.435821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13656.842385] Lustre: lustre-OST0000: new disk, initializing [13656.847736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13656.855644] Lustre: 352203:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13656.954592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13664.161246] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [13666.394234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13666.405217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13666.493779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13676.211283] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13723.106640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13723.124065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13723.126550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13728.441126] Lustre: server umount lustre-OST0000 complete [13733.353597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13733.358658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13733.382178] Lustre: Skipped 2 previous similar messages [13735.504825] Lustre: server umount lustre-MDT0000 complete [13738.472979] LustreError: 352023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13738.492182] LustreError: 352023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13740.761756] LustreError: 350225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788490783 with bad export cookie 11705534816542796142 [13740.772491] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13740.779993] LustreError: 350225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13741.310751] Lustre: server umount lustre-MDT0001 complete [13754.686262] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [13758.701617] Key type lgssc unregistered [13759.197883] LNet: 354377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13759.214356] LNetError: 354377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13759.263617] LNet: Removed LNI 192.168.204.112@tcp [13760.656357] Key type .llcrypt unregistered [13760.661865] Key type ._llcrypt unregistered [13868.546280] Key type ._llcrypt registered [13868.548887] Key type .llcrypt registered [13868.676436] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [13868.997552] Key type .llcrypt unregistered [13869.003862] Key type ._llcrypt unregistered [13959.334708] Key type ._llcrypt registered [13959.342507] Key type .llcrypt registered [13959.541617] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [13960.095509] Key type .llcrypt unregistered [13960.100253] Key type ._llcrypt unregistered [13994.850309] Key type ._llcrypt registered [13994.852776] Key type .llcrypt registered [13995.019458] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [13995.358978] Key type .llcrypt unregistered [13995.363422] Key type ._llcrypt unregistered [14029.627180] Key type ._llcrypt registered [14029.633027] Key type .llcrypt registered [14029.761629] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [14030.029972] Key type .llcrypt unregistered [14030.035358] Key type ._llcrypt unregistered [14113.231496] Key type ._llcrypt registered [14113.238297] Key type .llcrypt registered [14113.399269] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [14113.735275] Key type .llcrypt unregistered [14113.741711] Key type ._llcrypt unregistered [14147.751331] Key type ._llcrypt registered [14147.755814] Key type .llcrypt registered [14147.935856] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [14148.266289] Key type .llcrypt unregistered [14148.269331] Key type ._llcrypt unregistered [14180.110860] Key type ._llcrypt registered [14180.115137] Key type .llcrypt registered [14180.258843] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [14180.557492] Key type .llcrypt unregistered [14180.560820] Key type ._llcrypt unregistered [14212.915480] Key type ._llcrypt registered [14212.920829] Key type .llcrypt registered [14213.035715] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [14213.405387] Key type .llcrypt unregistered [14213.409785] Key type ._llcrypt unregistered [14251.659615] Key type ._llcrypt registered [14251.667237] Key type .llcrypt registered [14251.779792] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [14252.034364] Key type .llcrypt unregistered [14252.038555] Key type ._llcrypt unregistered [14288.536708] Key type ._llcrypt registered [14288.539487] Key type .llcrypt registered [14288.674756] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [14288.999240] Key type .llcrypt unregistered [14289.008617] Key type ._llcrypt unregistered [14356.713043] Key type ._llcrypt registered [14356.715852] Key type .llcrypt registered [14356.862778] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [14357.275312] Key type .llcrypt unregistered [14357.277123] Key type ._llcrypt unregistered [14388.875889] Key type ._llcrypt registered [14388.878145] Key type .llcrypt registered [14389.001097] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [14389.389580] Key type .llcrypt unregistered [14389.394458] Key type ._llcrypt unregistered [14421.826902] Key type ._llcrypt registered [14421.830544] Key type .llcrypt registered [14422.018837] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [14422.294592] Key type .llcrypt unregistered [14422.302285] Key type ._llcrypt unregistered [14483.575766] Key type ._llcrypt registered [14483.588506] Key type .llcrypt registered [14483.730979] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [14501.667612] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [14503.058919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14503.371884] alg: No test for adler32 (adler32-zlib) [14504.379744] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [14504.704771] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [14506.399603] Key type lgssc registered [14507.724154] Lustre: Echo OBD driver; http://www.lustre.org/ [14549.207927] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14549.228143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14550.634741] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14550.688845] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14550.811468] Lustre: tst250fs-MDT0000: new disk, initializing [14550.947235] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14550.981493] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14556.434636] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [14568.870083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14569.046659] Lustre: 374691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14569.161780] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14569.178821] Lustre: Skipped 1 previous similar message [14569.241338] Lustre: tst250fs-MDT0001: new disk, initializing [14569.355033] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14569.405986] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14569.420905] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14574.520034] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [14579.784625] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14587.170353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14587.569305] Lustre: tst250fs-OST0000: new disk, initializing [14587.575459] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14587.582828] Lustre: 376324:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14587.700836] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14590.576886] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14590.597572] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14590.794657] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14595.341958] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [14607.767015] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14607.954527] Lustre: tst250fs-OST0001: new disk, initializing [14607.970314] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14607.980211] Lustre: 377195:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14608.084873] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14614.132153] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14614.155174] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14614.297199] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14615.503409] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [14622.355553] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14624.824465] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14624.830460] Lustre: Skipped 1 previous similar message [14645.221619] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14645.222940] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14645.239033] Lustre: Skipped 1 previous similar message [14650.342722] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14650.353341] Lustre: Skipped 6 previous similar messages [14650.981175] Lustre: server umount tst250fs-MDT0000 complete [14656.041551] LustreError: 374684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788491698 with bad export cookie 9013431301593168931 [14656.048572] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14656.059150] LustreError: 374684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14656.596519] Lustre: server umount tst250fs-MDT0001 complete [14671.776494] Lustre: server umount tst250fs-OST0000 complete [14686.727651] Lustre: server umount tst250fs-OST0001 complete [14708.986670] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [14721.204684] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [14758.272240] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 23:16:38 (1788491798) [14760.722337] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14783.742556] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [14800.682284] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [14842.236924] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14511 sec ========= 23:18:02 (1788491882) [14844.814733] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:18:04 (1788491884) === [14848.810205] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:18:09 (1788491889) === [14892.331634] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [14896.034447] Key type lgssc unregistered [14896.483654] LNet: 386353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14896.491357] LNetError: 386353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14896.515145] LNet: Removed LNI 192.168.204.112@tcp [14897.524177] Key type .llcrypt unregistered [14897.526671] Key type ._llcrypt unregistered