[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 519659107 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002385] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.006855] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009170] LSM: Security Framework initializing [ 0.010068] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.013030] *** VALIDATE selinux *** [ 0.021266] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026485] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] *** VALIDATE tmpfs *** [ 0.031123] *** VALIDATE proc *** [ 0.032263] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037031] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041252] debug: unmapping init [mem 0xffffffff8d059000-0xffffffff8d060fff] [ 0.043895] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044702] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045029] ... version: 2 [ 0.046017] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048015] ... value mask: 0000ffffffffffff [ 0.049029] ... max period: 00007fffffffffff [ 0.050019] ... fixed-purpose events: 3 [ 0.051011] ... event mask: 000000070000000f [ 0.052283] rcu: Hierarchical SRCU implementation. [ 0.054425] smp: Bringing up secondary CPUs ... [ 0.055576] x86: Booting SMP configuration: [ 0.056027] .... node #0, CPUs: #1 #2 #3 [ 0.061008] smp: Brought up 1 node, 4 CPUs [ 0.063018] smpboot: Max logical packages: 1 [ 0.064013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.280701] node 0 deferred pages initialised in 214ms [ 0.287169] devtmpfs: initialized [ 0.288245] x86/mm: Memory block size: 128MB [ 0.292037] gcov: version magic: 0x41383552 [ 0.295399] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.299094] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.301327] pinctrl core: initialized pinctrl subsystem [ 0.306256] [ 0.306864] ************************************************************* [ 0.309016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.312013] ** ** [ 0.314016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.318016] ** ** [ 0.321018] ** This means that this kernel is built to expose internal ** [ 0.323012] ** IOMMU data structures, which may compromise security on ** [ 0.326015] ** your system. ** [ 0.331025] ** ** [ 0.334011] ** If you see this message and you are not debugging the ** [ 0.336018] ** kernel, report this immediately to your vendor! ** [ 0.339015] ** ** [ 0.341013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.344015] ************************************************************* [ 0.346619] NET: Registered protocol family 16 [ 0.349514] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.352071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.355092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.359093] cpuidle: using governor menu [ 0.360901] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.361730] PCI: Using configuration type 1 for base access [ 0.362154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.370157] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.373040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.377154] cryptd: max_cpu_qlen set to 1000 [ 0.381263] ACPI: Added _OSI(Module Device) [ 0.383021] ACPI: Added _OSI(Processor Device) [ 0.385015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.387024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.392818] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.399579] ACPI: Interpreter enabled [ 0.402080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.404022] ACPI: Using IOAPIC for interrupt routing [ 0.406166] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.410431] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.421090] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.423054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.426026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.429083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.434430] acpiphp: Slot [2] registered [ 0.436166] acpiphp: Slot [5] registered [ 0.438155] acpiphp: Slot [6] registered [ 0.440128] acpiphp: Slot [7] registered [ 0.441119] acpiphp: Slot [8] registered [ 0.443122] acpiphp: Slot [9] registered [ 0.444117] acpiphp: Slot [10] registered [ 0.447107] acpiphp: Slot [3] registered [ 0.448114] acpiphp: Slot [4] registered [ 0.450113] acpiphp: Slot [11] registered [ 0.452117] acpiphp: Slot [12] registered [ 0.453116] acpiphp: Slot [13] registered [ 0.455134] acpiphp: Slot [14] registered [ 0.456115] acpiphp: Slot [15] registered [ 0.458088] acpiphp: Slot [16] registered [ 0.459119] acpiphp: Slot [17] registered [ 0.461116] acpiphp: Slot [18] registered [ 0.462100] acpiphp: Slot [19] registered [ 0.464160] acpiphp: Slot [20] registered [ 0.465119] acpiphp: Slot [21] registered [ 0.466108] acpiphp: Slot [22] registered [ 0.468122] acpiphp: Slot [23] registered [ 0.470101] acpiphp: Slot [24] registered [ 0.472136] acpiphp: Slot [25] registered [ 0.474110] acpiphp: Slot [26] registered [ 0.475453] acpiphp: Slot [27] registered [ 0.477107] acpiphp: Slot [28] registered [ 0.479119] acpiphp: Slot [29] registered [ 0.480086] acpiphp: Slot [30] registered [ 0.482150] acpiphp: Slot [31] registered [ 0.483065] PCI host bridge to bus 0000:00 [ 0.485028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.487030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.490032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.492024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.495027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.498028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.500180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.503079] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.506364] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.517020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.522055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.525020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.527023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.530021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.533637] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.538134] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.540051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.544889] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.551021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.566023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.573021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.579715] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.592024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.601018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.625021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.636908] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.648028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.656020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.674047] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.686193] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.698020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.710022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.736022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.748664] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.758015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.766016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.788029] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.805033] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.812019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.817022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.840022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.855167] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.861018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.874023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.885018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.900083] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.902549] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.905414] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.908508] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.911268] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.917047] iommu: Default domain type: Passthrough [ 0.918663] SCSI subsystem initialized [ 0.924570] ACPI: bus type USB registered [ 0.931061] usbcore: registered new interface driver usbfs [ 0.936930] usbcore: registered new interface driver hub [ 0.940269] usbcore: registered new device driver usb [ 0.944222] pps_core: LinuxPPS API ver. 1 registered [ 0.947016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.956389] PTP clock support registered [ 0.961165] EDAC MC: Ver: 3.0.0 [ 0.967199] PCI: Using ACPI for IRQ routing [ 0.971789] NetLabel: Initializing [ 0.975024] NetLabel: domain hash size = 128 [ 0.980019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.986161] NetLabel: unlabeled traffic allowed by default [ 0.996017] vgaarb: loaded [ 1.007328] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.013016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.022435] clocksource: Switched to clocksource kvm-clock [ 1.355729] VFS: Disk quotas dquot_6.6.0 [ 1.357408] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.374187] *** VALIDATE ramfs *** [ 1.377251] *** VALIDATE hugetlbfs *** [ 1.394208] pnp: PnP ACPI init [ 1.402337] pnp: PnP ACPI: found 6 devices [ 1.507201] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.528914] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.531194] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.549708] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.552605] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.561774] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.575841] NET: Registered protocol family 2 [ 1.587861] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.608100] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.617243] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.626869] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.632778] TCP: Hash tables configured (established 65536 bind 65536) [ 1.640821] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.652728] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.666379] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.671771] NET: Registered protocol family 1 [ 1.677264] RPC: Registered named UNIX socket transport module. [ 1.681541] RPC: Registered udp transport module. [ 1.686734] RPC: Registered tcp transport module. [ 1.692433] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.708091] NET: Registered protocol family 44 [ 1.710060] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.712615] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.715096] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.728585] pci 0000:00:01.0: quirk_isa_dma_hangs+0x0/0x20 took 13171 usecs [ 1.733210] PCI: CLS 0 bytes, default 64 [ 1.735113] Unpacking initramfs... [ 5.100893] debug: unmapping init [mem 0xffff977b3cc54000-0xffff977b3ffbffff] [ 5.106888] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.109620] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.113195] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.307961] Initialise system trusted keyrings [ 6.309766] Key type blacklist registered [ 6.323751] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.359910] zbud: loaded [ 6.371976] *** VALIDATE nfs *** [ 6.373336] *** VALIDATE nfs4 *** [ 6.386465] pstore: using deflate compression [ 6.420126] Platform Keyring initialized [ 6.894367] NET: Registered protocol family 38 [ 6.900831] Key type asymmetric registered [ 6.911179] Asymmetric key parser 'x509' registered [ 6.917624] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.925200] io scheduler mq-deadline registered [ 6.929905] io scheduler kyber registered [ 6.937546] io scheduler bfq registered [ 6.943259] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.953637] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.964067] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.973226] ACPI: Power Button [PWRF] [ 6.990239] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.013753] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.040213] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.059552] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.110146] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.155604] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.211849] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.238798] Non-volatile memory driver v1.3 [ 7.246863] Linux agpgart interface v0.103 [ 7.333561] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 7.340266] vda: detected capacity change from 0 to 76681216 [ 7.379527] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.385903] vdb: detected capacity change from 0 to 1073741824 [ 7.423352] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.428934] vdc: detected capacity change from 0 to 2621440000 [ 7.471673] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.485885] vdd: detected capacity change from 0 to 2621440000 [ 7.545676] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.555933] vde: detected capacity change from 0 to 4294967296 [ 7.629735] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.636407] vdf: detected capacity change from 0 to 4294967296 [ 7.660137] libphy: Fixed MDIO Bus: probed [ 7.676893] usbcore: registered new interface driver usbserial_generic [ 7.683095] usbserial: USB Serial support registered for generic [ 7.688562] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.696844] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.701907] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.721990] mousedev: PS/2 mouse device common for all mice [ 7.734724] rtc_cmos 00:05: RTC can wake from S4 [ 7.745169] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.748030] rtc_cmos 00:05: registered as rtc0 [ 7.758720] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.773445] intel_pstate: CPU model not supported [ 7.781831] hid: raw HID events driver (C) Jiri Kosina [ 7.790766] usbcore: registered new interface driver usbhid [ 7.795468] usbhid: USB HID core driver [ 7.799720] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.812888] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.815688] drop_monitor: Initializing network drop monitor service [ 7.815836] Initializing XFRM netlink socket [ 7.816231] NET: Registered protocol family 10 [ 7.830064] Segment Routing with IPv6 [ 7.855863] NET: Registered protocol family 17 [ 7.860472] mpls_gso: MPLS GSO support [ 7.883461] RAS: Correctable Errors collector initialized. [ 7.897810] AVX version of gcm_enc/dec engaged. [ 7.901149] AES CTR mode by8 optimization enabled [ 8.296362] sched_clock: Marking stable (8296336759, 0)->(9327643729, -1031306970) [ 8.352726] registered taskstats version 1 [ 8.364853] Loading compiled-in X.509 certificates [ 8.368648] zswap: loaded using pool lzo/zbud [ 8.530615] Key type big_key registered [ 8.577410] Key type encrypted registered [ 8.582190] ima: No TPM chip found, activating TPM-bypass! [ 8.584189] ima: Allocated hash algorithm: sha1 [ 8.591309] ima: No architecture policies found [ 8.597604] evm: Initialising EVM extended attributes: [ 8.606564] evm: security.selinux [ 8.610255] evm: security.ima [ 8.616859] evm: security.capability [ 8.624060] evm: HMAC attrs: 0x1 [ 8.633915] rtc_cmos 00:05: setting system clock to 2026-09-04 03:24:09 UTC (1788492249) [ 8.651843] debug: unmapping init [mem 0xffffffff8e003000-0xffffffff8e1fffff] [ 8.659463] debug: unmapping init [mem 0xffffffff8cd82000-0xffffffff8d058fff] [ 8.673153] Write protecting the kernel read-only data: 28672k [ 8.688587] debug: unmapping init [mem 0xffffffff8b403000-0xffffffff8b5fffff] [ 8.698733] debug: unmapping init [mem 0xffffffff8bd14000-0xffffffff8bdfffff] [ 8.834833] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.845879] systemd[1]: Detected virtualization kvm. [ 8.849484] systemd[1]: Detected architecture x86-64. [ 8.851955] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.885723] systemd[1]: No hostname configured. [ 8.889121] systemd[1]: Set hostname to . [ 8.894703] random: systemd: uninitialized urandom read (16 bytes read) [ 8.901841] systemd[1]: Initializing machine ID from random generator. [ 9.011582] random: ln: uninitialized urandom read (6 bytes read) [ 9.216627] random: systemd: uninitialized urandom read (16 bytes read) [ 9.220755] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 9.228376] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 9.237145] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.928824] device-mapper: uevent: version 1.0.3 [ 10.932653] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.816326] virtio_net virtio0 ens2: renamed from eth0 [ 12.909182] random: fast init done [ 13.004328] scsi host0: ata_piix [ 13.061328] scsi host1: ata_piix [ 13.063317] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.066154] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.138417] random: crng init done [ 18.141597] random: 7 urandom warning(s) missed due to ratelimiting [ 20.498794] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 22.376465] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.538903] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.609274] SELinux: Disabled at runtime. [ 26.763695] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.784234] systemd[1]: Detected virtualization kvm. [ 26.789581] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.709784] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.715355] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.727540] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.737525] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.748857] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.761773] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.772460] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ 29.252066] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 30.395819] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.424149] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.591245] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.008179] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.208790] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 37.763487] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 38.535453] NFS: Registering the id_resolver key type [ 38.538511] Key type id_resolver registered [ ***] A start job is running for Configur…-only root support (9s / no limit)[ 38.547880] Key type id_legacy registered [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg414-server login: [ 89.260159] libcfs: loading out-of-tree module taints kernel. [ 89.326161] Key type ._llcrypt registered [ 89.333210] Key type .llcrypt registered [ 89.500897] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 112.242201] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 114.302763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.320808] alg: No test for adler32 (adler32-zlib) [ 115.697179] hrtimer: interrupt took 4847771 ns [ 116.133841] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 116.980301] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 118.928660] Key type lgssc registered [ 121.882969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 143.133680] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 191.863800] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 205.437970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 205.517089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.885478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 206.963110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 207.088223] Lustre: lustre-MDT0000: new disk, initializing [ 207.200631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.261554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 212.310772] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.918276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.075352] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 226.116502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 226.121800] Lustre: Skipped 1 previous similar message [ 226.323506] Lustre: lustre-MDT0001: new disk, initializing [ 226.458373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.496813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 226.524591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 231.488786] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.934661] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 248.011222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.201927] Lustre: lustre-OST0000: new disk, initializing [ 248.208481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 248.213241] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.257461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.955608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 252.967979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 253.095925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 254.667255] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.880784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 271.031660] Lustre: lustre-OST0001: new disk, initializing [ 271.035946] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 271.041269] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 271.120494] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 279.016759] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.645756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 279.666052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 279.806017] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 293.014150] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 301.630286] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 308.780249] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing check_logdir /tmp/testlogs/ [ 314.584457] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing yml_node [ 319.688239] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 322.853600] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 325.465418] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 326.971674] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 23:29:26 EDT 2026 [ 346.628799] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 348.573364] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 367.079223] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.081699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.102044] Lustre: Skipped 3 previous similar messages [ 367.124291] Lustre: Skipped 3 previous similar messages [ 372.054824] Lustre: server umount lustre-MDT0000 complete [ 377.129320] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788492617 with bad export cookie 1870980098822236347 [ 377.147060] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 377.320894] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 377.322946] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.334739] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 377.348395] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 382.434846] LustreError: 10362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 382.435286] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 382.456551] LustreError: 10362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 382.477217] Lustre: Skipped 2 previous similar messages [ 383.493517] Lustre: server umount lustre-MDT0001 complete [ 388.704531] Lustre: server umount lustre-OST0000 complete [ 394.242878] Lustre: server umount lustre-OST0001 complete [ 402.456976] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 413.302709] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 451.768508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.945691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 451.968227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 452.053363] Lustre: lustre-MDT0000: new disk, initializing [ 452.134069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.163392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 457.501215] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 468.246446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.548581] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 468.656219] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 468.663904] Lustre: Skipped 1 previous similar message [ 468.817956] Lustre: lustre-MDT0001: new disk, initializing [ 468.951484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 468.999532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 469.017897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 474.440892] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 484.035734] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 490.138401] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 496.464384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 496.741991] Lustre: lustre-OST0000: new disk, initializing [ 496.762797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 496.774718] Lustre: 17051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 496.939264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.024996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 498.051066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 498.201741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 503.853622] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 513.658920] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 520.020585] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 520.347785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 526.145951] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 526.446355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 528.872726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.885629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.885928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.892606] Lustre: Skipped 2 previous similar messages [ 528.915715] Lustre: Skipped 1 previous similar message [ 534.000789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 534.009762] Lustre: Skipped 1 previous similar message [ 534.959475] Lustre: server umount lustre-OST0000 complete [ 540.643958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 540.649993] LustreError: Skipped 1 previous similar message [ 540.654806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.228155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 544.238680] Lustre: Skipped 2 previous similar messages [ 546.121575] Lustre: server umount lustre-MDT0000 complete [ 549.350916] LustreError: 15096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.373782] LustreError: 15096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 550.287317] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788492791 with bad export cookie 1870980098822237880 [ 550.290604] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.303589] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 550.684834] Lustre: server umount lustre-MDT0001 complete [ 558.683167] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 23:33:17 (1788492797) [ 585.005912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.663848] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.755144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 590.819979] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.883937] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 600.834180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.060543] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.088356] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 606.983837] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 616.232948] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 622.667489] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 624.968305] Lustre: Failing over lustre-MDT0000 [ 625.308814] Lustre: server umount lustre-MDT0000 complete [ 627.042866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.069846] Lustre: Skipped 1 previous similar message [ 627.081781] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 627.100154] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 642.144183] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788492867/real 1788492867] req@ffff977baf75dc00 x1875370161899136/t0(0) o400->MGC192.168.204.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788492883 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 642.177863] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.744755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 652.455461] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x19f70e7cac1ff43e [ 652.475139] Lustre: MGC192.168.204.114@tcp: Connection restored to 0@lo (at 0@lo) [ 652.810583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 652.814282] Lustre: Skipped 1 previous similar message [ 652.895254] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 657.220323] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 657.891141] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 657.896764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 657.909299] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 667.596676] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 673.251351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 673.256256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 673.268295] Lustre: Skipped 1 previous similar message [ 673.277152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 678.637222] Lustre: server umount lustre-MDT0000 complete [ 681.465143] LustreError: 22228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 681.513616] LustreError: 22228:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 682.356970] LustreError: 19909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788492923 with bad export cookie 1870980098822239294 [ 682.367386] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 682.394679] LustreError: 19909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 682.902889] Lustre: server umount lustre-MDT0001 complete [ 692.093744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.317053] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 708.389842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.192752] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 722.724395] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 728.481627] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 731.367954] Lustre: Failing over lustre-MDT0000 [ 731.718358] Lustre: server umount lustre-MDT0000 complete [ 733.475144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.489234] Lustre: Skipped 2 previous similar messages [ 749.025760] LustreError: 23269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 749.058175] LustreError: 23269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 750.048181] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788492974/real 1788492974] req@ffff977a824f2a00 x1875370161929216/t0(0) o400->MGC192.168.204.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788492990 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 750.104804] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 752.850033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 759.329167] LustreError: 3644:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff977a824f1180 x1875370161932160/t0(0) o250->MGC192.168.204.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 760.047826] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 764.899258] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 764.912092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 764.912101] Lustre: Skipped 1 previous similar message [ 764.937422] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 765.183855] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 777.587347] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 783.336236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 783.358579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 783.365720] Lustre: Skipped 3 previous similar messages [ 788.362494] Lustre: server umount lustre-MDT0000 complete [ 792.661190] LustreError: 23249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788493033 with bad export cookie 1870980098822240232 [ 792.665539] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 792.675164] LustreError: 23249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 793.173663] Lustre: server umount lustre-MDT0001 complete [ 803.568289] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 23:37:23 (1788493043) [ 823.014345] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 835.186750] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 878.974906] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 892.244082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.644423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 892.687891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 892.780989] Lustre: lustre-MDT0000: new disk, initializing [ 892.931711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.939884] Lustre: Skipped 3 previous similar messages [ 892.968740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 897.657511] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 909.068231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.211518] Lustre: 29650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 909.519556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 914.027452] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 924.852918] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 931.688600] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 939.710028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 940.034967] Lustre: 31620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 941.715545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 941.756146] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 941.902092] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 941.918347] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 947.461910] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 959.997742] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 965.797409] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 988.138272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.145697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 988.153843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 988.180293] Lustre: Skipped 2 previous similar messages [ 988.224881] LustreError: Skipped 1 previous similar message [ 989.650230] Lustre: server umount lustre-OST0000 complete [ 996.835949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1000.353565] Lustre: server umount lustre-MDT0000 complete [ 1003.494344] LustreError: 29658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1003.510764] LustreError: 29658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 1004.792273] LustreError: 29641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788493245 with bad export cookie 1870980098822240638 [ 1004.794731] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1004.801252] LustreError: 29641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1005.480731] Lustre: server umount lustre-MDT0001 complete [ 1019.279734] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1023.833633] Key type lgssc unregistered [ 1024.362864] LNet: 33597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.390375] LNetError: 33597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.439853] LNet: Removed LNI 192.168.204.114@tcp [ 1025.778805] Key type .llcrypt unregistered [ 1025.783107] Key type ._llcrypt unregistered [ 1053.614702] Key type ._llcrypt registered [ 1053.619659] Key type .llcrypt registered [ 1053.746601] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1054.813888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1054.902091] alg: No test for adler32 (adler32-zlib) [ 1056.006755] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1056.274282] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1057.936833] Key type lgssc registered [ 1059.191892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1067.455069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1067.478344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.944884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.562806] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1078.246770] LustreError: 35008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1078.276959] LustreError: 35008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1083.191608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1083.376058] LustreError: 35009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1083.826665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1089.737675] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1097.278156] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1102.403350] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1108.874328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1109.200322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.270904] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1113.364369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1114.517563] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1122.593561] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1138.911031] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 23:42:57 (1788493377) [ 1141.466094] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1143.510433] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1150.129142] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 23:43:09 (1788493389) [ 1159.006775] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 23:43:18 (1788493398) [ 1160.408694] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1171.719590] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1171.724766] Lustre: Skipped 2 previous similar messages [ 1174.038439] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 23:43:33 (1788493413) [ 1176.311358] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1176.316984] Lustre: Skipped 2 previous similar messages [ 1186.281413] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1186.299484] Lustre: Skipped 4 previous similar messages [ 1201.890765] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 23:44:01 (1788493441) [ 1205.198637] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1209.004906] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1217.616172] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 23:44:17 (1788493457) [ 1219.681252] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1219.687566] Lustre: Skipped 6 previous similar messages [ 1232.582217] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 23:44:32 (1788493472) [ 1248.738813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1248.750734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.772109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1251.810062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1251.818738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1251.837921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.889463] Lustre: Skipped 1 previous similar message [ 1253.963034] Lustre: server umount lustre-OST0000 complete [ 1262.049368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.055914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1262.067563] Lustre: Skipped 1 previous similar message [ 1262.094132] Lustre: Skipped 1 previous similar message [ 1264.406149] Lustre: server umount lustre-MDT0000 complete [ 1267.171254] LustreError: 35009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.189746] LustreError: 35009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1268.626444] LustreError: 37927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788493509 with bad export cookie 12118094331666738672 [ 1268.636162] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1268.651034] LustreError: 37927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1269.199476] Lustre: server umount lustre-MDT0001 complete [ 1280.830535] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1283.793921] Key type lgssc unregistered [ 1284.145215] LNet: 42861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.166634] LNetError: 42861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1284.186896] LNet: Removed LNI 192.168.204.114@tcp [ 1285.137264] Key type .llcrypt unregistered [ 1285.138984] Key type ._llcrypt unregistered [ 1306.457584] Key type ._llcrypt registered [ 1306.460876] Key type .llcrypt registered [ 1306.565957] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 1323.968576] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1325.200472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1325.395022] alg: No test for adler32 (adler32-zlib) [ 1326.521232] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1326.789972] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1328.514700] Key type lgssc registered [ 1329.922773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1367.637132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1367.687347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.998234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1369.030670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1369.119939] Lustre: lustre-MDT0000: new disk, initializing [ 1369.215664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1369.240661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1374.124631] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1384.880555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1385.026064] Lustre: 46013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1385.103108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1385.106327] Lustre: Skipped 1 previous similar message [ 1385.195625] Lustre: lustre-MDT0001: new disk, initializing [ 1385.319819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1385.378441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1385.401383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1390.078947] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1400.813696] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1406.749358] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1414.872610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1415.190414] Lustre: lustre-OST0000: new disk, initializing [ 1415.199348] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1415.225497] Lustre: 47983:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1415.337018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.998439] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1424.457532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1424.475175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1424.539597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1432.057504] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1437.073361] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1437.303541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1442.710142] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1442.932482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1445.348487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1445.351428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.360124] LustreError: Skipped 1 previous similar message [ 1445.379412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1450.466044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1450.484437] Lustre: Skipped 2 previous similar messages [ 1455.587580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1455.603136] Lustre: Skipped 1 previous similar message [ 1459.168279] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1459.557863] Lustre: server umount lustre-OST0000 complete [ 1465.829207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1465.831761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1465.868491] Lustre: Skipped 2 previous similar messages [ 1465.886257] Lustre: Skipped 1 previous similar message [ 1470.230369] Lustre: server umount lustre-MDT0000 complete [ 1475.563699] LustreError: 46007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788493716 with bad export cookie 14395629063511619859 [ 1475.576491] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1475.586773] LustreError: 46007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1475.950699] Lustre: server umount lustre-MDT0001 complete [ 1490.116369] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 23:48:48 (1788493728) [ 1498.917954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1499.635327] LustreError: 50445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1499.655078] LustreError: 50445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1499.792016] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1505.124376] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1505.249793] LustreError: 50446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.349605] LustreError: 50445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1514.474613] LustreError: 50446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1514.835616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.232227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1520.178614] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1528.040681] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1534.762464] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1542.054911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1542.476218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1549.097107] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1557.634362] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1566.157856] LustreError: 52973:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1567.373116] Lustre: Modifying parameter general.timeout=1 in log params [ 1568.667590] Lustre: Modifying parameter general.timeout=2 in log params [ 1571.418470] Lustre: Modifying parameter general.timeout=5 in log params [ 1571.426656] Lustre: Skipped 2 previous similar messages [ 1575.641734] Lustre: Modifying parameter general.timeout=10 in log params [ 1575.653437] Lustre: Skipped 4 previous similar messages [ 1584.042653] Lustre: Modifying parameter general.timeout=19 in log params [ 1584.049395] Lustre: Skipped 8 previous similar messages [ 1600.252571] Lustre: Modifying parameter general.timeout=35 in log params [ 1600.258896] Lustre: Skipped 15 previous similar messages [ 1632.740552] Lustre: Modifying parameter general.timeout=69 in log params [ 1632.744691] Lustre: Skipped 33 previous similar messages [ 1701.591572] Lustre: Disabling parameter general.timeout= in log params [ 1701.597346] Lustre: Skipped 60 previous similar messages [ 1705.416992] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 23:52:24 (1788493944) [ 1708.368255] Lustre: 59936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1710.304700] Lustre: 60034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1714.199788] Lustre: 60233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1714.212593] Lustre: 60233:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1718.805368] Lustre: 60448:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1718.819774] Lustre: 60448:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1750.508667] Lustre: 61807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1750.546930] LustreError: 61807:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1755.659159] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 23:53:14 (1788493994) [ 1771.491137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1771.510561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.536154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1775.926859] Lustre: server umount lustre-MDT0000 complete [ 1778.154258] LustreError: 50446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1778.165154] LustreError: 50446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1779.596625] LustreError: 50426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788494020 with bad export cookie 14395629063511620741 [ 1779.606207] LustreError: 50426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1779.609035] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1779.990949] Lustre: server umount lustre-MDT0001 complete [ 1794.761971] Lustre: server umount lustre-OST0000 complete [ 1824.352931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1824.594470] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1824.871268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.579643] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1838.108945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1838.195624] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1838.225687] Lustre: 64340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1843.297338] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1850.575590] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1855.950852] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1863.064623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1863.493563] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1863.962185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1863.967682] Lustre: Skipped 1 previous similar message [ 1871.875983] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1879.724453] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1881.651897] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1881.660289] Lustre: Skipped 23 previous similar messages [ 1881.736684] Lustre: 66905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1890.785601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1890.796543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.826338] Lustre: Skipped 2 previous similar messages [ 1890.840672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1890.853642] Lustre: Skipped 3 previous similar messages [ 1895.395289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1895.402552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1896.472989] Lustre: server umount lustre-OST0000 complete [ 1907.275868] Lustre: server umount lustre-MDT0000 complete [ 1911.268354] LustreError: 64352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1911.294361] LustreError: 64352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1912.417313] LustreError: 66907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788494153 with bad export cookie 14395629063511623940 [ 1912.426817] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1912.435054] LustreError: 66907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1912.760382] Lustre: server umount lustre-MDT0001 complete [ 1926.542860] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1930.459710] Key type lgssc unregistered [ 1930.747450] LNet: 68117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.760274] LNetError: 68117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1930.787250] LNet: Removed LNI 192.168.204.114@tcp [ 1932.001761] Key type .llcrypt unregistered [ 1932.004788] Key type ._llcrypt unregistered [ 1958.489878] Key type ._llcrypt registered [ 1958.492567] Key type .llcrypt registered [ 1958.697990] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1960.130089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1960.195904] alg: No test for adler32 (adler32-zlib) [ 1961.292409] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1961.507952] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1963.152330] Key type lgssc registered [ 1964.446817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1973.651406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1973.699594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.265196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.750680] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1984.492705] LustreError: 69528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1984.529376] LustreError: 69528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1989.615899] LustreError: 69529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1990.579768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.225585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1996.452949] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2005.239650] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2011.063517] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2019.189515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2019.920746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.083303] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2035.588725] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2043.458677] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2055.652501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.678799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.683805] Lustre: Skipped 1 previous similar message [ 2055.711852] Lustre: Skipped 2 previous similar messages [ 2059.806792] Lustre: server umount lustre-MDT0000 complete [ 2060.780492] LustreError: 69523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.796684] LustreError: 69523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2063.800363] LustreError: 69509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788494304 with bad export cookie 13749515467927183163 [ 2063.816555] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2063.822418] LustreError: 69509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2064.320317] Lustre: server umount lustre-MDT0001 complete [ 2079.023377] Lustre: server umount lustre-OST0000 complete [ 2109.711411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.215231] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2110.308948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2110.610874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.318937] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2127.062081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.268889] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2127.336968] Lustre: 74400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2128.157725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2133.653175] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2143.460484] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2151.302445] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2159.383453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2159.718367] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2160.404515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.949381] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2176.831954] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2179.102419] Lustre: 76978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2181.175397] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2191.340484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2191.343729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.354833] LustreError: Skipped 1 previous similar message [ 2191.357906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2191.398613] Lustre: Skipped 2 previous similar messages [ 2193.985572] Lustre: server umount lustre-OST0000 complete [ 2199.522766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2199.546831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.577923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2199.586671] Lustre: Skipped 1 previous similar message [ 2201.571051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2201.598595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2201.609235] Lustre: Skipped 1 previous similar message [ 2206.692981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2206.702162] Lustre: Skipped 1 previous similar message [ 2212.320474] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2212.548692] Lustre: server umount lustre-MDT0000 complete [ 2216.612599] LustreError: 74393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788494457 with bad export cookie 13749515467927184003 [ 2216.621499] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2216.626879] LustreError: 74393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2217.056661] Lustre: server umount lustre-MDT0001 complete [ 2230.297275] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2233.316745] Key type lgssc unregistered [ 2233.630966] LNet: 78201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2233.638365] LNetError: 78201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2233.650936] LNet: Removed LNI 192.168.204.114@tcp [ 2234.637238] Key type .llcrypt unregistered [ 2234.639103] Key type ._llcrypt unregistered [ 2263.204173] Key type ._llcrypt registered [ 2263.206412] Key type .llcrypt registered [ 2263.304965] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2264.411660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2264.486961] alg: No test for adler32 (adler32-zlib) [ 2265.522825] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2265.756851] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2267.464250] Key type lgssc registered [ 2268.452268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2277.857290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2277.883907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2279.529440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2284.595023] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2290.150620] LustreError: 79612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2290.173102] LustreError: 79612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2294.014274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.685097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2299.079711] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2307.231395] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2313.124881] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2320.152936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2320.926766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.223563] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2338.642901] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2346.528332] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2347.882269] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2349.365101] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2351.539704] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2351.542782] Lustre: Skipped 1 previous similar message [ 2355.994566] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2356.006248] Lustre: Skipped 4 previous similar messages [ 2364.047705] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2364.055225] Lustre: Skipped 7 previous similar messages [ 2381.002936] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2381.009922] Lustre: Skipped 14 previous similar messages [ 2413.314620] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2413.322432] Lustre: Skipped 28 previous similar messages [ 2460.833712] Lustre: 87730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2464.471773] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 00:05:03 (1788494703) [ 2466.837938] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2469.080753] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 00:05:08 (1788494708) [ 2474.467393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2474.479513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2474.511474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2477.541164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2477.544888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2477.558189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.026409] Lustre: server umount lustre-OST0000 complete [ 2487.791256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.798203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2487.815865] Lustre: Skipped 1 previous similar message [ 2487.844992] Lustre: Skipped 2 previous similar messages [ 2492.896890] LustreError: 79613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.907486] LustreError: 79613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2493.291608] Lustre: server umount lustre-MDT0000 complete [ 2497.432239] LustreError: 84793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788494738 with bad export cookie 11464269952069246940 [ 2497.445798] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2497.460969] LustreError: 84793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2497.781651] Lustre: server umount lustre-MDT0001 complete [ 2514.114910] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2518.606357] Key type lgssc unregistered [ 2519.038300] LNet: 88979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2519.051823] LNetError: 88979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2519.095771] LNet: Removed LNI 192.168.204.114@tcp [ 2520.577217] Key type .llcrypt unregistered [ 2520.584766] Key type ._llcrypt unregistered [ 2526.565350] Key type ._llcrypt registered [ 2526.569737] Key type .llcrypt registered [ 2526.721050] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2531.894059] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules [ 2532.957086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2533.092589] alg: No test for adler32 (adler32-zlib) [ 2534.322991] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2534.624471] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2536.344830] Key type lgssc registered [ 2537.894147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2539.519317] LustreError: 90184:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2556.140721] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2561.070758] LustreError: 90184:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2565.103419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2565.122911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.576341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2571.162728] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2576.866285] LustreError: 91204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.887250] LustreError: 91204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2580.897561] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 00:07:00 (1788494820) [ 2581.985612] LustreError: 91205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.458736] LustreError: 91236:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2585.475460] LustreError: 91236:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2585.484771] LustreError: 91236:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2585.924731] Lustre: server umount lustre-MDT0000 complete [ 2601.674732] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2604.722890] Key type lgssc unregistered [ 2605.059238] LNet: 92729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2605.065974] LNetError: 92729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2605.095217] LNet: Removed LNI 192.168.204.114@tcp [ 2606.023617] Key type .llcrypt unregistered [ 2606.029643] Key type ._llcrypt unregistered [ 2623.177789] Key type ._llcrypt registered [ 2623.179851] Key type .llcrypt registered [ 2623.283366] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2624.422212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2624.487217] alg: No test for adler32 (adler32-zlib) [ 2625.609539] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2625.889668] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2627.608183] Key type lgssc registered [ 2628.776912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2636.698624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2636.714236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2638.203955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2642.660933] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2647.523973] LustreError: 93946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.565088] LustreError: 93946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2650.826464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.534558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2656.105401] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2663.784929] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2669.306758] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2676.556043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2677.634769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2683.783436] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2694.129196] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2745.022539] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 00:09:44 (1788494984) [ 2746.551594] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2748.438726] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 00:09:48 (1788494988) [ 2753.506356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2753.513341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.544592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.529423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.533734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.893346] Lustre: server umount lustre-MDT0000 complete [ 2759.663035] LustreError: 93942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2759.691902] LustreError: 93942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2763.039033] LustreError: 93927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788495003 with bad export cookie 4054580130170086244 [ 2763.053143] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2763.416728] Lustre: server umount lustre-MDT0001 complete [ 2777.907660] Lustre: server umount lustre-OST0000 complete [ 2789.170666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2789.619996] LustreError: 98247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2789.831767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2794.823990] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2794.980917] LustreError: 98248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.118016] LustreError: 98248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2803.091059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.787637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2808.738549] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2816.928526] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2823.247298] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2836.087564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2836.291943] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2836.303892] LustreError: 98235:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2836.337901] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2836.347336] LustreError: 100330:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2836.355415] LustreError: 100330:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2836.362170] LustreError: 100330:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2836.431365] Lustre: server umount lustre-OST0000 complete [ 2836.441626] LustreError: 100330:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2845.010700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2845.142055] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2845.148280] LustreError: 98237:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2845.156645] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2845.162799] LustreError: 100793:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2845.167627] LustreError: 100793:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2845.172570] LustreError: 100793:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2845.237537] Lustre: server umount lustre-OST0000 complete [ 2845.241802] LustreError: 100793:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2854.431340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2854.587708] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2854.592537] Lustre: Found index 0 for lustre-OST0000, updating log [ 2854.599554] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2854.677958] Lustre: lustre-OST0000: new disk, initializing [ 2854.685081] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2854.692649] Lustre: 101329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2854.994776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2860.833756] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2864.720264] Lustre: Failing over lustre-OST0000 [ 2864.939124] Lustre: server umount lustre-OST0000 complete [ 2870.242944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2870.249659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2870.266304] Lustre: Skipped 1 previous similar message [ 2870.273927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2870.281982] Lustre: Skipped 2 previous similar messages [ 2873.827628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2873.834797] Lustre: Skipped 2 previous similar messages [ 2878.954692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2878.964663] Lustre: Skipped 1 previous similar message [ 2879.038289] Lustre: server umount lustre-MDT0000 complete [ 2882.339156] LustreError: 101328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788495123 with bad export cookie 4054580130170088988 [ 2882.339650] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2882.349355] LustreError: 101328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2882.696418] Lustre: server umount lustre-MDT0001 complete [ 2892.317735] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 00:12:11 (1788495131) [ 2909.533848] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2920.218380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2920.501632] LustreError: 103674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2920.583428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.835579] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2932.942967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2937.364806] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2940.072070] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2946.074366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2946.705249] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2952.592348] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2957.923629] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2957.941324] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2957.961032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2960.393512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2960.517873] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2960.599820] Lustre: lustre-OST0001: new disk, initializing [ 2960.604629] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2960.614093] Lustre: 105989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2960.900567] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 2960.920291] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2960.924771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2960.925710] Lustre: Skipped 2 previous similar messages [ 2964.351446] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 2964.379772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2964.401655] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2964.518651] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2965.507439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2965.514607] LustreError: Skipped 1 previous similar message [ 2965.529262] Lustre: lustre-OST0000: Recovery over after 0:08, of 1 clients 1 recovered and 0 were evicted. [ 2965.530939] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2965.545887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2966.599052] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2973.552634] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2979.878510] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2983.997350] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.14@tcp (stopping) [ 2984.934605] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2984.942629] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2984.961394] Lustre: Skipped 1 previous similar message [ 2989.544948] Lustre: server umount lustre-MDT0001 complete [ 2991.076559] LustreError: 103671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2991.099264] LustreError: 103671:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2999.315332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2999.450642] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2999.456240] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2999.510547] Lustre: 103663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3001.529856] LustreError: 107579:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3001.539414] LustreError: 107579:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3001.556610] LustreError: 103667:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3001.667560] LustreError: 107589:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.114@tcp: cfg command failed: rc = -17 [ 3001.674488] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3001.674488] [ 3004.135601] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3005.423311] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3012.673533] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 00:14:12 (1788495252) [ 3028.520500] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3044.961545] Lustre: 109608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3067.406313] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3071.847043] Lustre: 110743:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3153.376828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3153.386052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.400797] Lustre: Skipped 2 previous similar messages [ 3153.409140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3153.416225] Lustre: Skipped 5 previous similar messages [ 3158.816284] Lustre: server umount lustre-MDT0000 complete [ 3159.013104] LustreError: 103669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.043222] LustreError: 103669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3162.855184] LustreError: 103655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788495403 with bad export cookie 4054580130170089919 [ 3162.856756] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3162.873410] LustreError: 103655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3163.247974] Lustre: server umount lustre-MDT0001 complete [ 3177.400721] Lustre: server umount lustre-OST0000 complete [ 3179.492606] Lustre: 93258:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788495404/real 1788495404] req@ffff977b8a5f1c00 x1875372792951424/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788495420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3179.554784] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3179.569916] Lustre: Skipped 3 previous similar messages [ 3181.633082] Lustre: server umount lustre-OST0001 complete [ 3187.637027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3199.615291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3210.362623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3221.735707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3244.277476] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3255.901828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3255.935885] Lustre: lustre-MDT0000: reset Object Index mappings [ 3256.563612] LustreError: 114693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3256.761598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3256.774387] Lustre: Skipped 1 previous similar message [ 3261.852169] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3270.740192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3270.766419] Lustre: lustre-MDT0001: reset Object Index mappings [ 3275.850837] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3279.068189] Lustre: 115841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3286.188660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3293.100699] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3295.240667] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3301.367631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3303.832990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3310.094033] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3310.100470] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3313.428492] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3323.112409] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3326.807977] Lustre: 117718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3376.223169] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:20:15 (1788495615) [ 3402.224568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.231418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3402.239384] Lustre: Skipped 3 previous similar messages [ 3402.254480] Lustre: Skipped 6 previous similar messages [ 3406.977810] Lustre: server umount lustre-MDT0000 complete [ 3407.349988] LustreError: 114694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3407.376032] LustreError: 114694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 3411.146133] LustreError: 114675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788495652 with bad export cookie 4054580130170137036 [ 3411.161938] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3417.891784] Lustre: server umount lustre-MDT0001 complete [ 3428.161436] Lustre: server umount lustre-OST0000 complete [ 3439.486587] Lustre: server umount lustre-OST0001 complete [ 3446.382289] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 3456.127988] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3492.276852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.556892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3492.599695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3492.694965] Lustre: lustre-MDT0000: new disk, initializing [ 3492.754565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3492.759394] Lustre: Skipped 3 previous similar messages [ 3492.768716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3497.744632] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3504.889510] Lustre: server umount lustre-MDT0000 complete [ 3514.254417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.546627] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3514.558183] Lustre: 121987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3514.566255] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3519.140765] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3523.828668] Lustre: server umount lustre-MDT0000 complete [ 3532.608834] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:22:52 (1788495772) [ 3534.236764] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3536.586750] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:22:55 (1788495775) [ 3555.328779] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3565.342518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.755354] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3579.655866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3579.803209] Lustre: 124037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3579.909285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3579.915403] Lustre: Skipped 1 previous similar message [ 3580.041675] Lustre: lustre-MDT0001: new disk, initializing [ 3580.199172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3580.217163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3586.068604] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3592.117869] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3601.444228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3601.822795] Lustre: lustre-OST0000: new disk, initializing [ 3601.828672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3601.845478] Lustre: 125615:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3603.407713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3603.427840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3603.654371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3608.152671] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3619.744834] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3619.922493] Lustre: lustre-OST0001: new disk, initializing [ 3619.926542] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3619.932354] Lustre: 126487:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3622.113036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3622.129310] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3622.256609] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3626.733205] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3636.921397] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3640.427272] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4026.944364] Lustre: 132912:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4026.952384] Lustre: 132912:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4026.959983] Lustre: 132912:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4026.964981] Lustre: 132912:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4026.975869] Lustre: 132912:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4026.980935] Lustre: 132912:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4320.501558] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:35:59 (1788496559) [ 4338.148251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4338.170796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.185179] Lustre: Skipped 2 previous similar messages [ 4338.206169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4338.218525] Lustre: Skipped 4 previous similar messages [ 4342.144258] Lustre: server umount lustre-MDT0000 complete [ 4344.295487] LustreError: 126879:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.315572] LustreError: 126879:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4345.902370] LustreError: 124032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788496586 with bad export cookie 4054580130170204537 [ 4345.902900] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4345.912064] LustreError: 124032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4349.409361] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.424612] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4349.430068] Lustre: Skipped 4 previous similar messages [ 4349.438127] Lustre: Skipped 5 previous similar messages [ 4352.562505] Lustre: server umount lustre-MDT0001 complete [ 4362.420712] Lustre: server umount lustre-OST0000 complete [ 4373.074217] Lustre: server umount lustre-OST0001 complete [ 4381.162684] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 4392.980868] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4431.842350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.113503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4432.145718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4432.212756] Lustre: lustre-MDT0000: new disk, initializing [ 4432.280089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4432.287587] Lustre: Skipped 5 previous similar messages [ 4432.299892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4437.098337] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4448.153672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4448.280364] Lustre: 135970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4448.346141] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4448.350445] Lustre: Skipped 1 previous similar message [ 4448.433397] Lustre: lustre-MDT0001: new disk, initializing [ 4448.603502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4448.624850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4454.392099] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4464.884713] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4471.548638] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4478.945960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4479.340288] Lustre: lustre-OST0000: new disk, initializing [ 4479.344492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4479.356564] Lustre: 137938:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4479.454322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.458403] Lustre: Skipped 1 previous similar message [ 4481.041826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4481.063530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4481.228895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4486.197549] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4496.370935] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4506.124073] Lustre: lustre-MDD0000: changelog on [ 4509.095106] Lustre: lustre-MDD0001: changelog on [ 6433.227133] Lustre: lustre-MDD0001: changelog off [ 6436.194159] Lustre: 140093:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6437.451467] Lustre: lustre-MDD0000: changelog off [ 6447.074911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6447.096897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6447.125677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6447.133709] Lustre: Skipped 2 previous similar messages [ 6447.613726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6450.699508] Lustre: server umount lustre-OST0000 complete [ 6455.779618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6455.795933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6455.823083] Lustre: Skipped 1 previous similar message [ 6455.830922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6455.842780] Lustre: Skipped 1 previous similar message [ 6457.829294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6461.760184] Lustre: server umount lustre-MDT0000 complete [ 6462.946771] LustreError: 136848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6462.972559] LustreError: 136848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6467.043460] LustreError: 135960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788498707 with bad export cookie 4054580130171023663 [ 6467.056780] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6467.062818] LustreError: 135960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6467.636929] Lustre: server umount lustre-MDT0001 complete [ 6481.984610] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 6485.068910] Key type lgssc unregistered [ 6485.371724] LNet: 147089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6485.376892] LNetError: 147089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6486.439165] LNet: Removed LNI 192.168.204.114@tcp [ 6487.546149] Key type .llcrypt unregistered [ 6487.551612] Key type ._llcrypt unregistered [ 6510.572634] Key type ._llcrypt registered [ 6510.576888] Key type .llcrypt registered [ 6510.753229] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 6528.459425] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6529.611374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6529.637591] alg: No test for adler32 (adler32-zlib) [ 6530.786995] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 6531.064137] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 6532.763375] Key type lgssc registered [ 6534.042467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6573.878562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6573.920882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6575.156241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6575.198640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6575.251984] Lustre: lustre-MDT0000: new disk, initializing [ 6575.323418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6575.345456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6580.751191] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6592.749131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6592.843180] Lustre: 150290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6592.893926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6592.898158] Lustre: Skipped 1 previous similar message [ 6592.949532] Lustre: lustre-MDT0001: new disk, initializing [ 6593.051511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6593.084582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6593.100767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6598.404423] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6609.076329] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6615.219528] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6622.147283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6622.523315] Lustre: lustre-OST0000: new disk, initializing [ 6622.526477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6622.535426] Lustre: 152258:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6622.729473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6629.718238] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6631.485045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6631.496388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6631.603075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6639.913607] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6646.957308] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_module obdecho/obdecho [ 6649.375772] LustreError: 153263:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6649.399058] LustreError: 153263:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6651.932807] LustreError: 153359:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6667.748400] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.756477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6667.762918] Lustre: Skipped 2 previous similar messages [ 6667.771215] Lustre: Skipped 2 previous similar messages [ 6672.862104] Lustre: server umount lustre-MDT0000 complete [ 6676.803296] LustreError: 150279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788498917 with bad export cookie 7961155156188437336 [ 6676.811191] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6676.815921] LustreError: 150279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6677.162541] Lustre: server umount lustre-MDT0001 complete [ 6691.771889] Lustre: server umount lustre-OST0000 complete [ 6719.371235] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 6733.496595] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6776.810337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6777.077425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6777.137426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6777.254463] Lustre: lustre-MDT0000: new disk, initializing [ 6777.359080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6777.375703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6782.926698] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6794.787223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6794.917126] Lustre: 157078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6795.011922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6795.015636] Lustre: Skipped 1 previous similar message [ 6795.099783] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6795.152380] Lustre: lustre-MDT0001: new disk, initializing [ 6795.261756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6795.309348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6795.331313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6800.339987] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6812.477577] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6819.282321] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6825.862593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6826.225389] Lustre: lustre-OST0000: new disk, initializing [ 6826.231553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6826.241870] Lustre: 159044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6826.342391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6832.878654] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6836.358107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6836.373807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6836.564533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6842.966190] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6847.981977] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6848.328263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6853.781788] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6854.036039] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6857.184944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6857.187582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6857.197890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6857.224683] Lustre: Skipped 1 previous similar message [ 6862.320066] LustreError: 159030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6862.352689] LustreError: 159030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6862.438174] Lustre: server umount lustre-OST0000 complete [ 6866.914118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6866.923707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6866.943634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6866.950954] Lustre: Skipped 1 previous similar message [ 6872.546125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6872.563687] Lustre: Skipped 3 previous similar messages [ 6874.842661] Lustre: server umount lustre-MDT0000 complete [ 6877.667872] LustreError: 159051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6877.715113] LustreError: 159051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6879.122566] LustreError: 157070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788499119 with bad export cookie 7961155156188438295 [ 6879.135525] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6879.148386] LustreError: 157070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6879.736449] Lustre: server umount lustre-MDT0001 complete [ 6884.097304] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:18:43 (1788499123) [ 6886.151945] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6887.965611] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 01:18:47 (1788499127) [ 6896.903772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6897.622279] LustreError: 161458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6897.726291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6902.754151] LustreError: 161459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6903.502459] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6907.877235] LustreError: 161458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6914.681108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6921.530161] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6930.668767] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6938.237874] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6947.774787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6948.547442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6948.564294] Lustre: Skipped 1 previous similar message [ 6956.085155] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6965.161451] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6981.201108] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.14@tcp (stopping) [ 6982.113108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6982.126145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6982.162460] Lustre: Skipped 1 previous similar message [ 6983.367346] Lustre: server umount lustre-MDT0000 complete [ 6983.651519] LustreError: 161458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6983.675705] LustreError: 161458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6987.943212] LustreError: 161437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788499228 with bad export cookie 7961155156188439177 [ 6987.965456] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6987.970208] LustreError: 161437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6988.776576] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.805620] Lustre: Skipped 2 previous similar messages [ 6994.777829] Lustre: server umount lustre-MDT0001 complete [ 7004.417042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7019.875571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7020.144085] LustreError: 165324:0:(llog.c:1655:llog_backup()) MGC192.168.204.114@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7020.152821] Lustre: 165324:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.114@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7021.216732] LustreError: 165329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7021.242275] LustreError: 165329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7053.297586] LustreError: 165330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7053.321306] LustreError: 165330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 7080.929699] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7080.936959] Lustre: 165324:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7117.853168] LustreError: 165329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7117.891340] LustreError: 165329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 7121.928169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7126.614266] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7127.038867] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7136.404749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7148.518770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7164.402769] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7164.874741] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7174.876720] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7176.903528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7185.671593] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7187.928881] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7210.471187] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7210.479724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7210.582403] Lustre: server umount lustre-MDT0000 complete [ 7214.703398] LustreError: 161440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788499455 with bad export cookie 7961155156188445666 [ 7214.712987] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7214.724783] LustreError: 161440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7215.099633] Lustre: server umount lustre-MDT0001 complete [ 7229.794839] Lustre: server umount lustre-OST0000 complete [ 7255.401411] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 7266.701110] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7304.652403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7304.877374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7304.910504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7305.003482] Lustre: lustre-MDT0000: new disk, initializing [ 7305.096640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7305.100626] Lustre: Skipped 1 previous similar message [ 7305.114940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7310.003421] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7320.786316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7320.877970] Lustre: 170433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7321.160328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7325.769894] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7335.473648] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7342.195295] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7350.339524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7350.638879] Lustre: lustre-OST0000: new disk, initializing [ 7350.643752] Lustre: Skipped 1 previous similar message [ 7350.654794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7350.660646] Lustre: Skipped 2 previous similar messages [ 7350.666405] Lustre: 172401:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7357.248755] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7358.616492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7358.637151] Lustre: Skipped 1 previous similar message [ 7358.654973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7358.882645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7367.974725] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7374.076512] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7374.320732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7379.717345] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7380.041438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7384.559165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7384.563891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7384.587469] Lustre: Skipped 2 previous similar messages [ 7384.593550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7384.593561] Lustre: Skipped 4 previous similar messages [ 7384.614426] LustreError: Skipped 1 previous similar message [ 7396.320210] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7396.509334] Lustre: server umount lustre-OST0000 complete [ 7402.988725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7406.723905] Lustre: server umount lustre-MDT0000 complete [ 7410.147146] LustreError: 170440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7410.176914] LustreError: 170440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 7410.714056] LustreError: 171227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788499651 with bad export cookie 7961155156188462459 [ 7410.715582] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7410.722279] LustreError: 171227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7411.170503] Lustre: server umount lustre-MDT0001 complete [ 7420.839880] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 01:27:40 (1788499660) [ 7429.927752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7437.286899] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7449.221245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7455.180397] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7465.715394] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7473.349882] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7482.185594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7491.188704] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7501.781882] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7513.382646] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 01:29:12 (1788499752) [ 7520.738692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7520.751457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.777122] Lustre: Skipped 3 previous similar messages [ 7520.782436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7520.789693] Lustre: Skipped 8 previous similar messages [ 7524.823873] Lustre: server umount lustre-OST0000 complete [ 7535.384793] Lustre: server umount lustre-MDT0000 complete [ 7539.566630] LustreError: 174795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788499780 with bad export cookie 7961155156188463341 [ 7539.571814] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7539.573100] LustreError: 174795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7539.950869] Lustre: server umount lustre-MDT0001 complete [ 7556.994742] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 7561.253244] Key type lgssc unregistered [ 7561.823866] LNet: 178546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7561.834109] LNetError: 178546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7561.856441] LNet: Removed LNI 192.168.204.114@tcp [ 7562.730857] Key type .llcrypt unregistered [ 7562.733610] Key type ._llcrypt unregistered [ 7582.142785] Key type ._llcrypt registered [ 7582.145540] Key type .llcrypt registered [ 7582.308950] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7584.117362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7584.223209] alg: No test for adler32 (adler32-zlib) [ 7585.526448] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 7585.867098] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 7587.584211] Key type lgssc registered [ 7588.899694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7599.888959] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7599.948791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7615.776389] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7620.960301] LustreError: 179787:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 7626.080444] LustreError: 179787:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7626.104712] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7626.114761] LustreError: MGC192.168.204.114@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7626.130973] LustreError: 179787:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7626.136254] LustreError: 179787:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 7626.143324] LustreError: 179787:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7626.216831] Lustre: server umount lustre-OST0000 complete [ 7626.219275] LustreError: 179787:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7637.715682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7639.373503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7645.027734] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7649.766505] LustreError: 180287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7649.789524] LustreError: 180287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7654.888206] LustreError: 180286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7656.520028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7657.116676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7663.378129] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7672.763905] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7678.883318] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7687.340203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7687.689354] LustreError: 182104:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7687.707858] Lustre: 182104:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.114@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7687.923453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7695.040366] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7703.115091] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7706.603958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7706.622095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7706.638695] Lustre: Skipped 1 previous similar message [ 7706.661295] Lustre: Skipped 2 previous similar messages [ 7711.716108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7711.740339] Lustre: Skipped 2 previous similar messages [ 7712.031159] Lustre: server umount lustre-MDT0000 complete [ 7716.110619] LustreError: 181021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788499956 with bad export cookie 117568051276506138 [ 7716.111109] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7716.120310] LustreError: 181021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7716.842552] LustreError: 180283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7716.852100] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7716.888567] Lustre: Skipped 1 previous similar message [ 7716.896337] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7716.897734] LustreError: 181016:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e14ab57c x1875377993578112/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7717.169386] Lustre: server umount lustre-MDT0001 complete [ 7723.381878] Lustre: server umount lustre-OST0000 complete [ 7734.193684] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 7746.896173] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7793.803595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7794.052125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7794.110747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7794.221686] Lustre: lustre-MDT0000: new disk, initializing [ 7794.287604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7794.302935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7799.834938] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7812.071235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7812.295758] Lustre: 185630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7812.373229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7812.382893] Lustre: Skipped 1 previous similar message [ 7812.449735] Lustre: lustre-MDT0001: new disk, initializing [ 7812.564786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7812.607407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7812.628329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7818.158337] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7828.734943] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7834.439548] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7841.744521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7842.157875] Lustre: lustre-OST0000: new disk, initializing [ 7842.168505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7842.182449] Lustre: 187598:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7842.354306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7849.127870] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7849.562908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7849.570812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7849.726315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7858.868071] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7863.774907] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7863.994654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7869.737511] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7869.919746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7875.553196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7875.554025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7875.570265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7875.590553] Lustre: Skipped 1 previous similar message [ 7875.609065] Lustre: Skipped 1 previous similar message [ 7878.531317] Lustre: server umount lustre-OST0000 complete [ 7884.257660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7884.275185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7884.289506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7889.575681] Lustre: server umount lustre-MDT0000 complete [ 7890.915564] LustreError: 187605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7890.945080] LustreError: 187605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7893.555663] LustreError: 185621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788500134 with bad export cookie 117568051276506852 [ 7893.557124] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7893.575403] LustreError: 185621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7893.816643] Lustre: server umount lustre-MDT0001 complete [ 7904.127685] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 01:35:43 (1788500143) [ 7924.540479] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 7927.829271] Key type lgssc unregistered [ 7928.323117] LNet: 190545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7928.339281] LNetError: 190545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7928.387771] LNet: Removed LNI 192.168.204.114@tcp [ 7929.845210] Key type .llcrypt unregistered [ 7929.851555] Key type ._llcrypt unregistered [ 7956.753871] Key type ._llcrypt registered [ 7956.760286] Key type .llcrypt registered [ 7956.977964] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7959.816148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7959.866304] alg: No test for adler32 (adler32-zlib) [ 7961.088862] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 7961.645486] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 7963.504166] Key type lgssc registered [ 7965.526161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7976.253313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7976.319915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7977.887533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7985.222893] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7987.200060] LustreError: 191762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7987.231663] LustreError: 191762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7992.290616] LustreError: 191763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7995.888534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7996.737639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8003.118864] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8011.995527] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8020.362337] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8030.484361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8030.912058] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8030.921694] LustreError: 193627:0:(llog.c:1696:llog_backup()) MGC192.168.204.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8030.936518] Lustre: 193627:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.114@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8031.179779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8039.352648] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8048.964687] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8052.708023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8052.712253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8052.730486] Lustre: Skipped 2 previous similar messages [ 8052.752348] Lustre: Skipped 2 previous similar messages [ 8057.850971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8057.860773] Lustre: Skipped 2 previous similar messages [ 8062.947344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8062.960852] Lustre: Skipped 2 previous similar messages [ 8066.016910] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8066.301512] Lustre: server umount lustre-MDT0000 complete [ 8068.070479] LustreError: 191762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8068.098190] LustreError: 191762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8071.026194] LustreError: 193654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788500311 with bad export cookie 2552272640974255471 [ 8071.036818] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8071.045543] LustreError: 193654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8071.432156] Lustre: server umount lustre-MDT0001 complete [ 8085.400868] Lustre: server umount lustre-OST0000 complete [ 8097.914000] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 8113.467537] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8158.827118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8159.242405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8159.322468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8159.448225] Lustre: lustre-MDT0000: new disk, initializing [ 8159.590710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8159.633696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8166.425643] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8179.434085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8179.634761] Lustre: 197185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8179.727589] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8179.730294] Lustre: Skipped 1 previous similar message [ 8179.733961] LustreError: 197965:0:(llog.c:1696:llog_backup()) MGC192.168.204.114@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8179.742406] LustreError: 197965:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8179.746826] Lustre: 197965:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.114@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8179.772511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8179.775148] Lustre: Skipped 1 previous similar message [ 8179.881297] Lustre: lustre-MDT0001: new disk, initializing [ 8180.025210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8180.102101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8180.130982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8185.866796] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8196.706616] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8204.219099] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8213.889835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8214.217351] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8214.222486] Lustre: Skipped 1 previous similar message [ 8214.228542] LustreError: 199128:0:(llog.c:1696:llog_backup()) MGC192.168.204.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8214.240961] LustreError: 199128:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8214.253448] Lustre: 199128:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.114@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8214.274484] Lustre: 199128:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8214.325564] Lustre: lustre-OST0000: new disk, initializing [ 8214.329286] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8214.336681] Lustre: 199151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8214.506562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8223.432610] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8223.863304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8223.875837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8223.960707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8234.044300] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8239.255902] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8239.519594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8245.237116] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8245.531269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8249.829946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8249.842659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8249.852473] Lustre: Skipped 1 previous similar message [ 8249.853452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8249.879493] LustreError: Skipped 1 previous similar message [ 8254.955030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8254.968689] Lustre: Skipped 2 previous similar messages [ 8262.112126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8262.370984] Lustre: server umount lustre-OST0000 complete [ 8267.237431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8267.253175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8267.299491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8267.307541] Lustre: Skipped 2 previous similar messages [ 8270.309370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8273.016465] Lustre: server umount lustre-MDT0000 complete [ 8275.426932] LustreError: 197193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8275.449711] LustreError: 197193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8277.405053] LustreError: 197177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788500518 with bad export cookie 2552272640974256192 [ 8277.407371] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8277.415598] LustreError: 197177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8277.952928] Lustre: server umount lustre-MDT0001 complete [ 8288.417443] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 01:42:07 (1788500527) [ 8308.862721] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8320.641047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8321.089799] LustreError: 202281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8321.123110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8325.904852] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8330.725143] LustreError: 202282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8330.755378] LustreError: 202282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8336.530176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8341.981473] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8345.331407] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8353.553446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8353.945257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8353.952855] Lustre: Skipped 1 previous similar message [ 8362.291820] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8372.957838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8373.223988] Lustre: lustre-OST0001: new disk, initializing [ 8373.229292] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8373.235734] Lustre: 204590:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8380.887282] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8383.026918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8383.037698] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8383.131687] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8392.410529] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8405.800460] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8409.093365] Lustre: DEBUG MARKER: ADD OST3 [ 8415.345915] loop: module loaded [ 8415.420714] loop0: detected capacity change from 0 to 204800000 [ 8415.795080] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8415.828213] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8415.870311] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8423.769568] Lustre: DEBUG MARKER: STOP OST3 [ 8424.047505] loop0: detected capacity change from 0 to 204800000 [ 8431.103685] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8431.342603] Lustre: lustre-OST0003: new disk, initializing [ 8431.350757] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8431.364747] Lustre: 206643:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8431.507742] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8431.514831] Lustre: Skipped 1 previous similar message [ 8432.903118] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8433.088543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8433.111945] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8433.167652] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8434.421792] Lustre: Failing over lustre-OST0003 [ 8434.792283] Lustre: server umount lustre-OST0003 complete [ 8434.983442] LustreError: 206764:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8437.833580] LustreError: 205056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8437.858723] LustreError: 205056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8438.243445] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8438.251141] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8438.269622] Lustre: Skipped 1 previous similar message [ 8439.169719] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8448.258397] Lustre: DEBUG MARKER: START OST3 again [ 8456.725277] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8457.065974] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8458.283640] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8458.639807] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8458.641269] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8465.366527] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8487.984488] Lustre: Failing over lustre-OST0003 [ 8488.204337] Lustre: server umount lustre-OST0003 complete [ 8489.447753] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8489.455758] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8489.463280] LustreError: 203764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8489.475143] Lustre: Skipped 1 previous similar message [ 8489.495851] LustreError: 203764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8510.948543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8510.961954] LustreError: Skipped 1 previous similar message [ 8510.972954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8510.992786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8511.007034] Lustre: Skipped 2 previous similar messages [ 8515.406811] Lustre: server umount lustre-MDT0000 complete [ 8519.638860] LustreError: 204589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788500760 with bad export cookie 2552272640974257074 [ 8519.646728] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8519.658636] LustreError: 204589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8526.070639] Lustre: server umount lustre-MDT0001 complete [ 8530.251461] Lustre: server umount lustre-OST0000 complete [ 8534.423538] Lustre: server umount lustre-OST0001 complete [ 8541.972384] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 8551.376809] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8594.438575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8594.683649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8594.749759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8594.848308] Lustre: lustre-MDT0000: new disk, initializing [ 8594.906773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8594.911861] Lustre: Skipped 1 previous similar message [ 8594.921945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8600.249943] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8612.335808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8612.521755] Lustre: 212065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8612.639213] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8612.648895] Lustre: Skipped 1 previous similar message [ 8612.816677] Lustre: lustre-MDT0001: new disk, initializing [ 8613.055208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8613.065314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8619.191735] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8629.639180] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8635.839644] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8644.168788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8644.508802] Lustre: 214035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8645.602479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8645.624566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8645.783783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8652.281588] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8662.527264] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8667.864474] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8668.124978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8673.204248] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8673.430163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8676.328543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8676.332203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.343073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8676.343080] Lustre: Skipped 8 previous similar messages [ 8676.373506] Lustre: Skipped 6 previous similar messages [ 8682.379503] Lustre: server umount lustre-OST0000 complete [ 8693.299444] Lustre: server umount lustre-MDT0000 complete [ 8696.811990] LustreError: 212078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8696.832528] LustreError: 212078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 8697.869705] LustreError: 212058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788500938 with bad export cookie 2552272640974259356 [ 8697.872826] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8697.879982] LustreError: 212058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8698.467357] Lustre: server umount lustre-MDT0001 complete [ 8702.541542] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 01:49:01 (1788500941) [ 8720.469867] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 8732.728529] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8778.024531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8778.369946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8778.417906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8778.428493] Lustre: Skipped 1 previous similar message [ 8778.541547] Lustre: lustre-MDT0000: new disk, initializing [ 8778.549210] Lustre: Skipped 1 previous similar message [ 8778.744920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8783.938746] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8795.360970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8795.548535] Lustre: 218647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8795.869322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8801.232533] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8812.333727] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8818.932343] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8826.141155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8826.355088] Lustre: 220619:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8828.458369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8828.649669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8833.071265] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8842.407584] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8847.354096] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8847.593179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8852.177837] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8852.416305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8854.500271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8854.502869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8854.514563] LustreError: Skipped 1 previous similar message [ 8854.534918] Lustre: Skipped 2 previous similar messages [ 8854.540076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8854.551287] Lustre: Skipped 7 previous similar messages [ 8860.370449] Lustre: server umount lustre-OST0000 complete [ 8870.986802] Lustre: server umount lustre-MDT0000 complete [ 8874.553610] LustreError: 218641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788501115 with bad export cookie 2552272640974260245 [ 8874.556931] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8874.561872] LustreError: 218641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8874.939926] Lustre: server umount lustre-MDT0001 complete [ 8884.129161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.655901] LustreError: 222833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8884.755068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8884.765680] Lustre: Skipped 5 previous similar messages [ 8889.900594] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8900.647481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8906.422961] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8915.894704] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8923.994392] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8934.532253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8942.325864] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8951.504452] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8970.224466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8970.240925] LustreError: Skipped 1 previous similar message [ 8974.042632] Lustre: server umount lustre-OST0000 complete [ 8985.571210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8985.589239] Lustre: Skipped 11 previous similar messages [ 8986.181314] Lustre: server umount lustre-MDT0000 complete [ 8990.981323] LustreError: 222815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788501231 with bad export cookie 2552272640974261127 [ 8990.988664] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8990.992088] LustreError: 222815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8991.329581] Lustre: server umount lustre-MDT0001 complete [ 9093.309037] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 9097.264842] Key type lgssc unregistered [ 9097.563788] LNet: 226523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9097.573859] LNetError: 226523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9098.600059] LNet: Removed LNI 192.168.204.114@tcp [ 9099.625325] Key type .llcrypt unregistered [ 9099.628492] Key type ._llcrypt unregistered [ 9127.001354] Key type ._llcrypt registered [ 9127.004025] Key type .llcrypt registered [ 9127.190436] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 9144.299488] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9146.007778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9146.101968] alg: No test for adler32 (adler32-zlib) [ 9147.243428] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 9147.600636] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 9149.320187] Key type lgssc registered [ 9150.542390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9190.134070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9190.162875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9191.495870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9191.528299] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9191.616518] Lustre: lustre-MDT0000: new disk, initializing [ 9191.781428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9191.800960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9196.452464] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9208.889996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9209.022709] Lustre: 229874:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9209.113285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9209.117186] Lustre: Skipped 1 previous similar message [ 9209.217063] Lustre: lustre-MDT0001: new disk, initializing [ 9209.316875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9209.366713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9209.385648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9214.056925] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9224.275936] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9231.807469] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9240.547626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9241.032742] Lustre: lustre-OST0000: new disk, initializing [ 9241.039900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9241.064942] Lustre: 231844:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9241.211753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9248.090654] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9248.874609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9248.894257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9249.016238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9259.331833] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9264.245600] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9264.449716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9269.567933] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9269.898246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9274.852167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9274.854297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9274.865365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9274.909084] Lustre: Skipped 1 previous similar message [ 9278.329799] Lustre: server umount lustre-OST0000 complete [ 9285.091675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9285.106527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9285.107039] Lustre: Skipped 1 previous similar message [ 9285.126032] Lustre: Skipped 2 previous similar messages [ 9290.209364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9290.215438] Lustre: Skipped 1 previous similar message [ 9294.307518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9294.324214] Lustre: Skipped 1 previous similar message [ 9296.352304] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9296.588103] Lustre: server umount lustre-MDT0000 complete [ 9300.771820] LustreError: 230670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788501541 with bad export cookie 13587868192073156752 [ 9300.781820] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9300.786139] LustreError: 230670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9301.029714] Lustre: server umount lustre-MDT0001 complete [ 9322.078795] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9333.444973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9333.755718] LustreError: 234777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9333.769910] LustreError: 234777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9333.820511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9338.852207] LustreError: 234778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9339.444671] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9343.971399] LustreError: 234777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9348.918979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9349.090579] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9349.194745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9354.193555] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9357.464758] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9366.012457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9366.486409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9373.295566] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9383.396715] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9383.603831] Lustre: lustre-OST0001: new disk, initializing [ 9383.608924] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9383.616319] Lustre: 237085:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9383.716842] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9389.195034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9389.213435] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9389.327700] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9390.131417] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9401.618238] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9408.641128] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9423.339956] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 02:01:02 (1788501662) [ 9430.508800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9430.511881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9430.541860] Lustre: Skipped 2 previous similar messages [ 9430.571502] Lustre: Skipped 3 previous similar messages [ 9432.431261] Lustre: server umount lustre-MDT0000 complete [ 9435.621303] LustreError: 234777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9435.639700] LustreError: 234777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9436.446140] LustreError: 234758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788501677 with bad export cookie 13587868192073157634 [ 9436.463045] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9436.477259] LustreError: 234758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9437.127639] Lustre: server umount lustre-MDT0001 complete [ 9452.221521] Lustre: server umount lustre-OST0000 complete [ 9466.838929] Lustre: server umount lustre-OST0001 complete [ 9474.331592] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 9485.678302] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9522.557714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9522.874551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9522.919929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9523.002619] Lustre: lustre-MDT0000: new disk, initializing [ 9523.088867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9523.104187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9528.828358] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9539.510126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9539.637576] Lustre: 241045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9539.695474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9539.699235] Lustre: Skipped 1 previous similar message [ 9539.785477] Lustre: lustre-MDT0001: new disk, initializing [ 9539.986144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9540.004379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9544.983209] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9554.393319] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9559.828995] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9567.073444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9567.387452] Lustre: lustre-OST0000: new disk, initializing [ 9567.397172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9567.403155] Lustre: 243012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9569.103421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9569.125502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9569.277119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9573.890922] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9583.076672] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9588.275913] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9588.622695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9593.739139] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9593.992373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9599.969695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9599.981050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9599.987299] LustreError: Skipped 1 previous similar message [ 9599.995703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9600.017913] Lustre: Skipped 2 previous similar messages [ 9600.046352] Lustre: Skipped 1 previous similar message [ 9610.208395] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9610.384674] Lustre: server umount lustre-OST0000 complete [ 9615.330823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9615.344486] Lustre: Skipped 1 previous similar message [ 9620.182797] Lustre: server umount lustre-MDT0000 complete [ 9620.455371] LustreError: 241053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9620.471761] LustreError: 241053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9624.262729] LustreError: 241037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788501865 with bad export cookie 13587868192073159762 [ 9624.268934] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9624.276217] LustreError: 241037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9624.585250] Lustre: server umount lustre-MDT0001 complete [ 9633.105959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9633.741797] LustreError: 245229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9633.870176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9633.879062] Lustre: Skipped 2 previous similar messages [ 9638.078977] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9647.273352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9652.588320] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9661.133488] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9667.193246] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9673.900158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9680.901210] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9688.110519] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9832.420104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9832.439836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9832.465770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9832.474255] Lustre: Skipped 6 previous similar messages [ 9836.524558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9838.335741] Lustre: server umount lustre-OST0000 complete [ 9848.617440] Lustre: server umount lustre-MDT0000 complete [ 9851.880096] LustreError: 245230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9851.909214] LustreError: 245230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9852.345331] LustreError: 245209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788502093 with bad export cookie 13587868192073160651 [ 9852.370332] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9852.370500] LustreError: 245209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9852.741505] Lustre: server umount lustre-MDT0001 complete [ 9957.606836] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 9961.286696] Key type lgssc unregistered [ 9961.767932] LNet: 248838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9961.793540] LNetError: 248838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9961.814948] LNet: Removed LNI 192.168.204.114@tcp [ 9963.052719] Key type .llcrypt unregistered [ 9963.054186] Key type ._llcrypt unregistered [ 9990.164164] Key type ._llcrypt registered [ 9990.169789] Key type .llcrypt registered [ 9990.275721] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [10006.789933] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10007.952184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10008.024024] alg: No test for adler32 (adler32-zlib) [10009.040646] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [10009.414096] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [10011.104112] Key type lgssc registered [10012.676044] Lustre: Echo OBD driver; http://www.lustre.org/ [10061.782457] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10074.184792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10074.236702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10075.527736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10075.624943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10075.698696] Lustre: lustre-MDT0000: new disk, initializing [10075.849552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10075.887460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10080.577302] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10091.659900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.819487] Lustre: 252903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10091.912418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10091.918085] Lustre: Skipped 1 previous similar message [10092.031257] Lustre: lustre-MDT0001: new disk, initializing [10092.209391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10092.263635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10092.278825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10097.884608] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10103.279415] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10110.287452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10110.575967] Lustre: lustre-OST0000: new disk, initializing [10110.582075] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10110.591115] Lustre: 254537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10110.684596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10113.999945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10114.022545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10114.081627] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10117.134960] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10128.257272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10128.455155] Lustre: lustre-OST0001: new disk, initializing [10128.458890] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10128.465148] Lustre: 255409:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10128.562986] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10135.327960] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10137.198602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10137.208813] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10137.302514] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10146.806914] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10157.557897] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10168.291984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10168.301491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10168.319747] Lustre: Skipped 2 previous similar messages [10168.340362] Lustre: Skipped 3 previous similar messages [10170.341063] Lustre: server umount lustre-MDT0000 complete [10174.014587] LustreError: 253707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788502414 with bad export cookie 4161354222564261390 [10174.028166] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10174.035786] LustreError: 253707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10174.361507] Lustre: server umount lustre-MDT0001 complete [10187.978407] Lustre: server umount lustre-OST0000 complete [10201.712434] Lustre: server umount lustre-OST0001 complete [10222.090312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10232.911475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10241.866458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10252.102757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10273.953970] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10285.213926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10285.234893] Lustre: lustre-MDT0000: reset Object Index mappings [10285.782145] LustreError: 260423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10285.799102] LustreError: 260423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10285.876389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10290.910583] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10291.173267] LustreError: 260424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10295.272901] LustreError: 260423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10300.396229] LustreError: 260424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10300.470571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10300.490174] Lustre: lustre-MDT0001: reset Object Index mappings [10300.958469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10305.919544] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10308.732633] Lustre: 261566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10316.318365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10323.593683] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10326.895242] LustreError: 261923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10332.027143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10332.618579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10333.058139] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10333.071503] Lustre: Skipped 1 previous similar message [10338.308870] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10340.754638] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10348.600796] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10352.014315] Lustre: 263437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10357.219745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10357.232862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10357.265492] Lustre: Skipped 1 previous similar message [10357.275361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10358.681767] Lustre: server umount lustre-MDT0000 complete [10358.756484] LustreError: 260424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10358.763479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10358.782733] LustreError: 260424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10358.813961] Lustre: Skipped 1 previous similar message [10362.599500] LustreError: 261568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788502603 with bad export cookie 4161354222564268180 [10362.611963] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10363.094965] Lustre: server umount lustre-MDT0001 complete [10376.393385] Lustre: server umount lustre-OST0000 complete [10389.447732] Lustre: server umount lustre-OST0001 complete [10408.766490] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 02:17:28 (1788502648) [10426.225741] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [10439.199772] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10479.473398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10479.928726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10479.971332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10480.150066] Lustre: lustre-MDT0000: new disk, initializing [10480.246945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10480.290843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10486.146628] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10499.604459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10499.716425] Lustre: 267696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10499.766347] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10499.771860] Lustre: Skipped 1 previous similar message [10499.838931] Lustre: lustre-MDT0001: new disk, initializing [10500.037368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10500.062919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10505.521718] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10515.691596] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10522.022475] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10529.684116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10529.974329] Lustre: lustre-OST0000: new disk, initializing [10529.989459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10529.997310] Lustre: 269664:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10531.094632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10531.109834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10531.255411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10536.603546] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10547.984772] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10553.922751] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10554.275687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10561.243529] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10561.618454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10567.157253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10567.174324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10567.177565] LustreError: Skipped 1 previous similar message [10567.185039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10567.241244] Lustre: Skipped 2 previous similar messages [10570.566681] Lustre: server umount lustre-OST0000 complete [10576.866592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10576.889091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10576.910266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10576.924269] Lustre: Skipped 1 previous similar message [10581.284489] Lustre: server umount lustre-MDT0000 complete [10582.497194] LustreError: 267701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10582.532355] LustreError: 267701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10585.733753] LustreError: 267687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788502826 with bad export cookie 4161354222564270273 [10585.740925] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10585.753978] LustreError: 267687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10586.540260] Lustre: server umount lustre-MDT0001 complete [10612.202109] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10624.883659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10625.542386] LustreError: 272621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10625.616441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10625.621181] Lustre: Skipped 2 previous similar messages [10631.375216] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10643.084413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10648.716845] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10651.876503] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10659.831904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10667.106370] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10676.223540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10676.444630] Lustre: lustre-OST0001: new disk, initializing [10676.449174] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10676.454582] Lustre: 274930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10678.032526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10678.040186] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10678.166345] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10682.836767] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10693.240091] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10696.837952] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10702.316530] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10702.408633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10713.634829] Lustre: server umount lustre-MDT0000 complete [10714.083647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10714.095269] LustreError: 272618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10714.099211] Lustre: Skipped 4 previous similar messages [10714.124581] LustreError: 272618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10717.407678] LustreError: 272603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788502958 with bad export cookie 4161354222564271155 [10717.418283] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10717.421515] LustreError: 272603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10717.893823] Lustre: server umount lustre-MDT0001 complete [10732.330376] Lustre: server umount lustre-OST0000 complete [10736.097085] Lustre: 250409:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788502960/real 1788502960] req@ffff977b9f4e0380 x1875380535529472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788502976 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10736.130407] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10737.183764] Lustre: server umount lustre-OST0001 complete [10758.253238] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10770.279058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10771.080125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10771.088820] Lustre: Skipped 3 previous similar messages [10776.874808] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10788.920546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10794.359275] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10797.496716] Lustre: 279410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10804.716345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10812.714077] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10815.633700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10821.720115] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10827.265765] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10828.032141] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10835.487284] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10838.917429] Lustre: 281286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10844.225731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10844.350550] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10857.930442] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 02:24:57 (1788503097) [10873.325900] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10873.348079] Lustre: Skipped 2 previous similar messages [10873.349421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10873.373101] Lustre: Skipped 5 previous similar messages [10879.539592] Lustre: server umount lustre-MDT0000 complete [10883.247767] LustreError: 279789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788503124 with bad export cookie 4161354222564277525 [10883.267647] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10883.269437] LustreError: 279789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10883.561035] LustreError: 278264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10883.561977] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10883.594562] LustreError: 278264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10883.611802] Lustre: Skipped 5 previous similar messages [10889.748038] Lustre: server umount lustre-MDT0001 complete [10900.413672] Lustre: server umount lustre-OST0000 complete [10904.901487] Lustre: server umount lustre-OST0001 complete [10912.272628] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [10927.179862] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10983.485280] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10995.901687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10996.208163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10996.280429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10996.427415] Lustre: lustre-MDT0000: new disk, initializing [10996.597774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11003.060741] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11015.810424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11015.938957] Lustre: 285816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11016.238486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11020.902501] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11033.536974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11033.864232] Lustre: 287452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11034.047604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11034.058755] Lustre: Skipped 5 previous similar messages [11035.755082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11040.619613] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11051.655212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11051.891456] Lustre: 288324:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11053.164294] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11053.184552] Lustre: Skipped 1 previous similar message [11053.328543] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11058.970596] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11068.458410] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11071.674875] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11071.685898] Lustre: Skipped 1 previous similar message [11088.028509] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 02:28:46 (1788503326) [11090.955716] Lustre: server umount lustre-MDT0000 complete [11092.964782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11092.981818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11093.001357] Lustre: Skipped 3 previous similar messages [11095.050704] LustreError: 287451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788503335 with bad export cookie 4161354222564285463 [11095.054992] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11095.066128] LustreError: 287451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11097.513958] Lustre: server umount lustre-MDT0001 complete [11112.154753] Lustre: server umount lustre-OST0000 complete [11126.346974] Lustre: server umount lustre-OST0001 complete [11135.637964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11140.633094] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11141.610231] LustreError: 290573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11141.629805] LustreError: 290573:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [11149.200454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11154.120189] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11162.871607] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11169.416597] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11176.781841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11184.245611] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11192.168987] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11212.506774] loop1: detected capacity change from 0 to 204800000 [11212.799783] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11212.814783] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11212.828838] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11217.901810] loop1: detected capacity change from 0 to 204800000 [11223.883967] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11223.977176] LustreError: 290560:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs31712: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11223.991353] LustreError: 290560:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs31712 [11223.995694] LustreError: 290560:0:(mgs_handler.c:517:mgs_target_reg()) fs31712-MDT0001: Error checking target index: rc = -13 [11224.002337] LustreError: fs31712-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11224.008650] LustreError: 294293:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11224.013646] LustreError: 294293:0:(tgt_mount.c:2132:server_put_super()) no obd fs31712-MDT0001 [11224.017638] LustreError: 294293:0:(tgt_mount.c:116:server_deregister_mount()) fs31712-MDT0001 not registered [11224.052903] Lustre: server umount fs31712-MDT0001 complete [11224.055707] LustreError: 294293:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11233.731512] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11233.884198] Lustre: Setting parameter fs31712-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs31712-MDT0001 [11233.954644] Lustre: srv-fs31712-MDT0001: No data found on store. Initialize space. [11233.957457] Lustre: Skipped 4 previous similar messages [11234.072197] Lustre: fs31712-MDT0001: new disk, initializing [11234.075698] Lustre: Skipped 3 previous similar messages [11234.220924] Lustre: srv-fs31712-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11234.232712] Lustre: 294761:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs31712-MDT0001-mdtlov: can't create new llog: rc = -115 [11234.249147] LustreError: 294761:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs31712-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11239.929682] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11247.413327] Lustre: server umount fs31712-MDT0001 complete [11261.037725] loop1: detected capacity change from 0 to 204800000 [11261.415843] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11266.832937] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11267.196792] Lustre: 296331:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs31712-MDT0001-mdtlov: can't create new llog: rc = -115 [11267.208894] LustreError: 296331:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs31712-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11272.862649] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11276.792411] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11278.366711] Lustre: server umount fs31712-MDT0001 complete [11290.205763] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.14@tcp (stopping) [11290.210991] Lustre: Skipped 2 previous similar messages [11292.134142] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11292.161758] Lustre: Skipped 4 previous similar messages [11294.758111] Lustre: server umount lustre-MDT0000 complete [11304.055045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11304.162151] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11308.933052] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11309.556882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11316.627735] loop1: detected capacity change from 0 to 204800000 [11322.215228] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11322.637987] Lustre: 298776:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs31712-MDT0001-mdtlov: can't create new llog: rc = -115 [11322.649983] LustreError: 298776:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs31712-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11327.523373] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11332.620357] Lustre: server umount fs31712-MDT0001 complete [11358.694738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11358.705861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11358.732268] Lustre: Skipped 3 previous similar messages [11364.374923] Lustre: server umount lustre-OST0000 complete [11376.582222] Lustre: server umount lustre-MDT0000 complete [11380.753903] LustreError: 290553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788503621 with bad export cookie 4161354222564288144 [11380.769168] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11380.775649] LustreError: 290553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11381.368542] Lustre: server umount lustre-MDT0001 complete [11392.748622] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [11395.670977] Key type lgssc unregistered [11396.043323] LNet: 301118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11396.051746] LNetError: 301118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11396.070416] LNet: Removed LNI 192.168.204.114@tcp [11397.147211] Key type .llcrypt unregistered [11397.148981] Key type ._llcrypt unregistered [11419.203631] Key type ._llcrypt registered [11419.208327] Key type .llcrypt registered [11419.322625] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [11420.445123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11420.606123] alg: No test for adler32 (adler32-zlib) [11421.767367] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [11422.015837] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [11423.656164] Key type lgssc registered [11424.922061] Lustre: Echo OBD driver; http://www.lustre.org/ [11432.671725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11432.686148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11434.101313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11438.362847] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11443.683972] LustreError: 302394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11443.704418] LustreError: 302394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11446.754976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11447.278819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11452.495597] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11461.037922] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11467.657495] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11475.298633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11475.699714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11478.758181] LustreError: 304228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11482.235420] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11484.146759] LustreError: 304409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11489.257486] LustreError: 304227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11491.558982] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11494.394082] LustreError: 304229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11494.438384] LustreError: 304229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11502.562553] LustreError: 304227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11502.587133] LustreError: 304227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11511.294860] loop1: detected capacity change from 0 to 204800000 [11511.535891] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11511.555034] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11511.576797] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11517.191118] loop1: detected capacity change from 0 to 204800000 [11520.045498] LustreError: 304409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11520.088875] LustreError: 304409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11524.354566] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11524.580919] Lustre: fs23720-OST0000: new disk, initializing [11524.588291] Lustre: srv-fs23720-OST0000: No data found on store. Initialize space. [11524.614759] Lustre: 306132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11524.730681] Lustre: fs23720-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11532.607279] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11541.964216] Lustre: server umount fs23720-OST0000 complete [11553.772668] LustreError: 304229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11553.786642] LustreError: 304229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11554.401842] loop1: detected capacity change from 0 to 204800000 [11558.897862] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11560.462135] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11560.776918] Lustre: fs23720-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11568.065945] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11572.217375] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11577.414217] loop2: detected capacity change from 0 to 204800000 [11577.615939] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11577.625983] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11577.634632] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11583.878575] loop2: detected capacity change from 0 to 204800000 [11589.665417] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11589.786286] LustreError: fs23720-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11589.802314] LustreError: fs23720-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11589.812572] LustreError: 309175:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11589.818661] LustreError: 309175:0:(tgt_mount.c:2132:server_put_super()) no obd fs23720-OST0001 [11589.825224] LustreError: 309175:0:(tgt_mount.c:116:server_deregister_mount()) fs23720-OST0001 not registered [11589.898492] Lustre: server umount fs23720-OST0001 complete [11589.905619] LustreError: 309175:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11601.771971] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11602.024526] Lustre: fs23720-OST0001: new disk, initializing [11602.030648] Lustre: srv-fs23720-OST0001: No data found on store. Initialize space. [11602.043736] Lustre: 309727:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11602.134053] Lustre: fs23720-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11609.031888] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11617.952829] Lustre: server umount fs23720-OST0000 complete [11618.281559] LustreError: 304229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11618.295854] LustreError: 304229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11628.048693] Lustre: server umount fs23720-OST0001 complete [11642.749531] loop1: detected capacity change from 0 to 204800000 [11649.262175] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11649.741867] Lustre: fs23720-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11658.402851] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11665.656541] loop2: detected capacity change from 0 to 204800000 [11672.039988] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11679.071775] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11689.166992] Lustre: server umount fs23720-OST0000 complete [11699.215773] Lustre: server umount fs23720-OST0001 complete [11710.458264] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11712.483472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11712.499128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11712.516627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11714.529359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11714.544738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11715.560165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11715.570104] Lustre: Skipped 1 previous similar message [11718.698752] Lustre: server umount lustre-OST0000 complete [11723.756557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11723.761742] LustreError: Skipped 1 previous similar message [11723.764688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11723.785786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11725.795783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11729.421794] Lustre: server umount lustre-MDT0000 complete [11733.234530] LustreError: 302375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788503974 with bad export cookie 3768473234442424428 [11733.241164] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11733.250866] LustreError: 302375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11733.715235] Lustre: server umount lustre-MDT0001 complete [11746.446790] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [11749.704291] Key type lgssc unregistered [11750.097812] LNet: 315650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11750.107969] LNetError: 315650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11750.137891] LNet: Removed LNI 192.168.204.114@tcp [11751.035182] Key type .llcrypt unregistered [11751.038254] Key type ._llcrypt unregistered [11785.931174] Key type ._llcrypt registered [11785.935224] Key type .llcrypt registered [11786.090704] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [11803.720497] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [11805.011100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11805.049889] alg: No test for adler32 (adler32-zlib) [11806.302628] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [11806.611548] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [11808.336174] Key type lgssc registered [11809.733397] Lustre: Echo OBD driver; http://www.lustre.org/ [11854.925466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11854.966446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11856.302149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11856.366623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11856.473653] Lustre: lustre-MDT0000: new disk, initializing [11856.570464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11856.593543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11861.073712] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11873.279421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11873.438759] Lustre: 319509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11873.540993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11873.556575] Lustre: Skipped 1 previous similar message [11873.687264] Lustre: lustre-MDT0001: new disk, initializing [11873.812804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11873.879402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11873.891920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11878.970481] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11888.313936] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11895.447798] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11904.461744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11904.759854] Lustre: lustre-OST0000: new disk, initializing [11904.763565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11904.770600] Lustre: 321478:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11904.865554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11911.234345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11911.243734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11911.392508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11911.809950] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [11921.125451] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11923.097305] Lustre: Mounted lustre-client - version 2.17.58_2_gbb63b80 [11924.209469] Lustre: Unmounted lustre-client [11933.668922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11933.690452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11938.790062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11938.804073] Lustre: Skipped 2 previous similar messages [11939.044630] Lustre: server umount lustre-OST0000 complete [11943.909132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11943.918027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11943.928481] Lustre: Skipped 2 previous similar messages [11943.941437] Lustre: Skipped 1 previous similar message [11949.025561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11949.044646] Lustre: Skipped 1 previous similar message [11954.154022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11954.160055] Lustre: Skipped 1 previous similar message [11957.216130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11957.438822] Lustre: server umount lustre-MDT0000 complete [11961.398110] LustreError: 319502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788504202 with bad export cookie 2366813620327369896 [11961.414885] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11961.417593] LustreError: 319502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11961.815681] Lustre: server umount lustre-MDT0001 complete [11973.287451] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [11975.587712] LNet: 1 local NIs in recovery (showing 1): 192.168.204.114@tcp [11976.279215] Key type lgssc unregistered [11976.654995] LNet: 323641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11976.667734] LNetError: 323641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12051.439514] LNet: Removed LNI 192.168.204.114@tcp [12052.525160] Key type .llcrypt unregistered [12052.527129] Key type ._llcrypt unregistered [12072.215341] Key type ._llcrypt registered [12072.219343] Key type .llcrypt registered [12072.391884] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [12087.853690] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [12088.959581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12089.043023] alg: No test for adler32 (adler32-zlib) [12090.190150] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [12090.396690] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [12092.120429] Key type lgssc registered [12093.102289] Lustre: Echo OBD driver; http://www.lustre.org/ [12127.326646] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:46:06 (1788504366) [12137.574742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12137.605901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12139.131920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12139.173616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12139.254081] Lustre: lustre-MDT0000: new disk, initializing [12139.325434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12139.343243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12144.652062] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12156.326492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12156.511488] Lustre: 326861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12156.603613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12156.612774] Lustre: Skipped 1 previous similar message [12156.741120] Lustre: lustre-MDT0001: new disk, initializing [12156.912484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12156.961515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12156.970642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12161.817412] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12172.219077] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12178.499219] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12186.442315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12187.048743] Lustre: lustre-OST0000: new disk, initializing [12187.060615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12187.083407] Lustre: 328828:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12187.213909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12194.841213] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12197.513754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12197.527702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12197.641815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12205.041453] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12223.457490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12223.466235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12223.483867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12223.501087] Lustre: Skipped 1 previous similar message [12226.567795] Lustre: server umount lustre-OST0000 complete [12233.697305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12233.699232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12233.731571] Lustre: Skipped 1 previous similar message [12233.770802] Lustre: Skipped 2 previous similar messages [12238.279685] Lustre: server umount lustre-MDT0000 complete [12242.096213] LustreError: 326853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788504482 with bad export cookie 8257431362121727875 [12242.098401] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12242.108077] LustreError: 326853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12242.420298] Lustre: server umount lustre-MDT0001 complete [12253.811379] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [12256.910168] Key type lgssc unregistered [12257.233644] LNet: 330737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12257.243959] LNetError: 330737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12258.279214] LNet: Removed LNI 192.168.204.114@tcp [12259.091889] Key type .llcrypt unregistered [12259.102376] Key type ._llcrypt unregistered [12279.187388] Key type ._llcrypt registered [12279.189559] Key type .llcrypt registered [12279.306384] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [12280.227803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12280.237408] alg: No test for adler32 (adler32-zlib) [12281.265890] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [12281.541893] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [12283.240183] Key type lgssc registered [12284.257083] Lustre: Echo OBD driver; http://www.lustre.org/ [12291.237331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12291.257476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12292.676291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12297.901207] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12302.821973] LustreError: 332006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12302.859215] LustreError: 332006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12307.941486] LustreError: 332007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12308.573855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12309.101786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12314.851621] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12323.595448] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12330.745880] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12340.473590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12341.015445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12347.824629] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12355.920888] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12371.820594] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12373.481877] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12378.095662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12378.102580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12378.107719] Lustre: Skipped 2 previous similar messages [12378.123636] Lustre: Skipped 2 previous similar messages [12383.202668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12383.217637] Lustre: Skipped 3 previous similar messages [12383.715808] Lustre: server umount lustre-MDT0000 complete [12387.750744] LustreError: 331988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788504628 with bad export cookie 12037319173519452509 [12387.756739] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12387.767712] LustreError: 331988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12388.331058] LustreError: 332003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12388.335062] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12388.374131] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12388.378124] Lustre: Skipped 1 previous similar message [12392.357057] LustreError: 332003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12392.361492] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12392.379710] LustreError: 332003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12392.407016] Lustre: Skipped 1 previous similar message [12397.486299] LustreError: 332742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12397.500231] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12397.516127] LustreError: 332742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12397.529351] Lustre: Skipped 1 previous similar message [12402.144261] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12402.478293] Lustre: server umount lustre-MDT0001 complete [12406.489775] Lustre: server umount lustre-OST0000 complete [12416.246834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12421.665892] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12425.101723] Lustre: 336208:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.114@tcp [12427.704267] Lustre: server umount MGS complete [12437.070950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12437.640114] LustreError: 336779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12437.664804] LustreError: 336779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12437.745442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12442.627729] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12450.794805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12451.176202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12455.535134] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12464.371681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12464.760687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12471.169639] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12478.363710] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12479.893063] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12485.928807] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12487.579187] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12492.774061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12492.782364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12492.801075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12492.820937] Lustre: Skipped 1 previous similar message [12498.714751] Lustre: server umount lustre-OST0000 complete [12503.011665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12503.035187] Lustre: Skipped 1 previous similar message [12508.863249] Lustre: server umount lustre-MDT0000 complete [12512.648241] LustreError: 336760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788504753 with bad export cookie 12037319173519453349 [12512.654795] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12512.662584] LustreError: 336760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12513.042550] Lustre: server umount lustre-MDT0001 complete [12522.923062] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [12526.206029] Key type lgssc unregistered [12526.553331] LNet: 340216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12526.560748] LNetError: 340216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12526.582481] LNet: Removed LNI 192.168.204.114@tcp [12527.480466] Key type .llcrypt unregistered [12527.482745] Key type ._llcrypt unregistered [12546.037721] Key type ._llcrypt registered [12546.040693] Key type .llcrypt registered [12546.179471] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [12562.516286] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [12563.659153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12563.718440] alg: No test for adler32 (adler32-zlib) [12564.759588] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [12565.040530] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [12566.736222] Key type lgssc registered [12567.887409] Lustre: Echo OBD driver; http://www.lustre.org/ [12601.914431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12601.939285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12603.185549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12603.223155] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12603.305131] Lustre: lustre-MDT0000: new disk, initializing [12603.369218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12603.389267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12608.488707] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12618.855889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12618.972467] Lustre: 343375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12619.031341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12619.036376] Lustre: Skipped 1 previous similar message [12619.086921] Lustre: lustre-MDT0001: new disk, initializing [12619.161913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12619.215121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12619.225594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12623.331941] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12632.821880] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12638.099874] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12644.635517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12644.829643] Lustre: lustre-OST0000: new disk, initializing [12644.832626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12644.838650] Lustre: 345341:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12644.911525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12650.452977] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12653.658921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12653.674503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12653.768521] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12658.695592] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12663.488740] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12663.661935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12667.666196] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12667.846393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12674.529757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12674.538048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12674.553490] LustreError: Skipped 1 previous similar message [12674.557982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12674.573106] Lustre: Skipped 1 previous similar message [12674.599764] Lustre: Skipped 1 previous similar message [12679.652979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12679.672657] Lustre: Skipped 1 previous similar message [12684.256151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12684.402705] Lustre: server umount lustre-OST0000 complete [12689.893525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12689.912299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12689.916170] Lustre: Skipped 1 previous similar message [12694.525667] Lustre: server umount lustre-MDT0000 complete [12698.070488] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12698.070088] LustreError: 343365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788504938 with bad export cookie 2973113843075574573 [12698.093346] LustreError: 343365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12698.473265] Lustre: server umount lustre-MDT0001 complete [12706.095695] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:55:45 (1788504945) [12719.760349] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [12728.500813] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [12758.158666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12758.355387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12758.376210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12758.430152] Lustre: lustre-MDT0000: new disk, initializing [12758.482086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12758.500660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12762.953454] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12773.069285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12773.193792] Lustre: 350102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12773.251987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12773.256628] Lustre: Skipped 1 previous similar message [12773.347791] Lustre: lustre-MDT0001: new disk, initializing [12773.488059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12773.550117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12773.569660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12778.235149] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12787.615828] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12792.893633] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12798.715923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12798.871684] Lustre: lustre-OST0000: new disk, initializing [12798.874333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12798.879195] Lustre: 352070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12798.953704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12804.197811] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [12808.247939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12808.253915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12808.396136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12813.253451] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12844.513709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12844.520482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12844.536443] Lustre: Skipped 2 previous similar messages [12844.543893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12844.550315] Lustre: Skipped 1 previous similar message [12849.637726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12849.645520] Lustre: Skipped 1 previous similar message [12850.467941] Lustre: server umount lustre-OST0000 complete [12854.758833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12854.773069] Lustre: Skipped 1 previous similar message [12859.876523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12859.885316] Lustre: Skipped 3 previous similar messages [12868.064144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12868.250780] Lustre: server umount lustre-MDT0000 complete [12870.117983] LustreError: 350107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12870.139814] LustreError: 350107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12872.018703] LustreError: 350904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788505112 with bad export cookie 2973113843075575448 [12872.020169] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12872.026416] LustreError: 350904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12872.362795] Lustre: server umount lustre-MDT0001 complete [12883.230461] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [12886.399656] Key type lgssc unregistered [12886.722782] LNet: 354221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12886.730655] LNetError: 354221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12886.742417] LNet: Removed LNI 192.168.204.114@tcp [12887.580720] Key type .llcrypt unregistered [12887.583794] Key type ._llcrypt unregistered [12976.257412] Key type ._llcrypt registered [12976.259034] Key type .llcrypt registered [12976.342909] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [12976.653310] Key type .llcrypt unregistered [12976.659783] Key type ._llcrypt unregistered [13051.741944] Key type ._llcrypt registered [13051.743474] Key type .llcrypt registered [13051.870289] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13052.239306] Key type .llcrypt unregistered [13052.246045] Key type ._llcrypt unregistered [13081.120209] Key type ._llcrypt registered [13081.121887] Key type .llcrypt registered [13081.193741] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13081.431323] Key type .llcrypt unregistered [13081.436106] Key type ._llcrypt unregistered [13108.384946] Key type ._llcrypt registered [13108.387511] Key type .llcrypt registered [13108.486592] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13108.789289] Key type .llcrypt unregistered [13108.798216] Key type ._llcrypt unregistered [13188.767403] Key type ._llcrypt registered [13188.772162] Key type .llcrypt registered [13188.895426] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13189.240546] Key type .llcrypt unregistered [13189.249452] Key type ._llcrypt unregistered [13219.653329] Key type ._llcrypt registered [13219.660237] Key type .llcrypt registered [13219.848436] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13220.160426] Key type .llcrypt unregistered [13220.165113] Key type ._llcrypt unregistered [13251.238866] Key type ._llcrypt registered [13251.241536] Key type .llcrypt registered [13251.340634] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13251.577589] Key type .llcrypt unregistered [13251.581747] Key type ._llcrypt unregistered [13282.914537] Key type ._llcrypt registered [13282.916918] Key type .llcrypt registered [13283.018649] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13283.289634] Key type .llcrypt unregistered [13283.303568] Key type ._llcrypt unregistered [13311.120117] Key type ._llcrypt registered [13311.122400] Key type .llcrypt registered [13311.240978] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13311.478413] Key type .llcrypt unregistered [13311.480552] Key type ._llcrypt unregistered [13342.070436] Key type ._llcrypt registered [13342.080904] Key type .llcrypt registered [13342.217629] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13342.583601] Key type .llcrypt unregistered [13342.589918] Key type ._llcrypt unregistered [13402.813801] Key type ._llcrypt registered [13402.816684] Key type .llcrypt registered [13402.899344] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13403.275299] Key type .llcrypt unregistered [13403.277716] Key type ._llcrypt unregistered [13430.191212] Key type ._llcrypt registered [13430.194597] Key type .llcrypt registered [13430.331371] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13430.614405] Key type .llcrypt unregistered [13430.618648] Key type ._llcrypt unregistered [13461.507587] Key type ._llcrypt registered [13461.511200] Key type .llcrypt registered [13461.587081] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13461.841400] Key type .llcrypt unregistered [13461.843702] Key type ._llcrypt unregistered [13512.360497] Key type ._llcrypt registered [13512.365408] Key type .llcrypt registered [13512.537290] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [13528.715563] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [13529.561457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13529.708271] alg: No test for adler32 (adler32-zlib) [13530.900698] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [13531.213595] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [13532.920375] Key type lgssc registered [13534.142525] Lustre: Echo OBD driver; http://www.lustre.org/ [13567.812635] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13567.858722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13569.105906] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13569.141824] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13569.218776] Lustre: tst250fs-MDT0000: new disk, initializing [13569.315647] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13569.331725] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13573.545361] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13583.693111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13583.768963] Lustre: 374442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13583.826381] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13583.831072] Lustre: Skipped 1 previous similar message [13583.900262] Lustre: tst250fs-MDT0001: new disk, initializing [13583.987885] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13584.027245] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13584.048907] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13588.927074] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13593.767417] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13600.662368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13600.902626] Lustre: tst250fs-OST0000: new disk, initializing [13600.909628] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13600.915886] Lustre: 376075:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13600.998379] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13605.118732] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13605.134962] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13605.253717] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13606.881201] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13617.111390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13617.300969] Lustre: tst250fs-OST0001: new disk, initializing [13617.304716] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13617.313578] Lustre: 376946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13617.442641] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13624.140926] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [13627.517685] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13627.525663] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13627.610821] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13629.590384] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13632.473641] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13632.479127] Lustre: Skipped 2 previous similar messages [13653.481775] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13653.485704] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13653.496234] Lustre: Skipped 2 previous similar messages [13653.508255] Lustre: Skipped 2 previous similar messages [13656.894795] Lustre: server umount tst250fs-MDT0000 complete [13660.230987] LustreError: 374434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788505901 with bad export cookie 4879073775668616494 [13660.234523] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13660.252080] LustreError: 374434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13660.745895] Lustre: server umount tst250fs-MDT0001 complete [13674.510661] Lustre: server umount tst250fs-OST0000 complete [13687.442706] Lustre: server umount tst250fs-OST0001 complete [13705.515680] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [13713.710651] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [13744.352901] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 03:13:03 (1788505983) [13745.806302] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13762.194778] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [13772.338188] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [13806.789823] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13478 sec ========= 03:14:06 (1788506046) [13808.739658] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:14:08 (1788506048) === [13812.878336] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:14:12 (1788506052) === [13858.494029] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [13861.701259] Key type lgssc unregistered [13862.048207] LNet: 386089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13862.055661] LNetError: 386089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13862.073800] LNet: Removed LNI 192.168.204.114@tcp [13862.870227] Key type .llcrypt unregistered [13862.874568] Key type ._llcrypt unregistered