[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 757252118 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003272] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.007852] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010138] LSM: Security Framework initializing [ 0.011068] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013065] *** VALIDATE selinux *** [ 0.023449] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030052] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032178] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034114] *** VALIDATE tmpfs *** [ 0.036254] *** VALIDATE proc *** [ 0.037228] *** VALIDATE cgroup *** [ 0.038016] *** VALIDATE cgroup2 *** [ 0.040203] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044010] Spectre V2 : User space: Vulnerable [ 0.045010] Speculative Store Bypass: Vulnerable [ 0.047783] debug: unmapping init [mem 0xffffffffb8859000-0xffffffffb8860fff] [ 0.050190] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051761] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052022] ... version: 2 [ 0.053014] ... bit width: 48 [ 0.054014] ... generic registers: 4 [ 0.055012] ... value mask: 0000ffffffffffff [ 0.056018] ... max period: 00007fffffffffff [ 0.057015] ... fixed-purpose events: 3 [ 0.058014] ... event mask: 000000070000000f [ 0.059308] rcu: Hierarchical SRCU implementation. [ 0.061470] smp: Bringing up secondary CPUs ... [ 0.062679] x86: Booting SMP configuration: [ 0.063026] .... node #0, CPUs: #1 #2 #3 [ 0.072017] smp: Brought up 1 node, 4 CPUs [ 0.074072] smpboot: Max logical packages: 1 [ 0.075124] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.118021] node 0 deferred pages initialised in 41ms [ 0.122228] devtmpfs: initialized [ 0.123335] x86/mm: Memory block size: 128MB [ 0.126047] gcov: version magic: 0x41383552 [ 0.128164] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.129075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.130301] pinctrl core: initialized pinctrl subsystem [ 0.131193] [ 0.132008] ************************************************************* [ 0.133012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.134019] ** ** [ 0.135016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.137018] ** ** [ 0.138019] ** This means that this kernel is built to expose internal ** [ 0.139012] ** IOMMU data structures, which may compromise security on ** [ 0.140016] ** your system. ** [ 0.141011] ** ** [ 0.142013] ** If you see this message and you are not debugging the ** [ 0.143012] ** kernel, report this immediately to your vendor! ** [ 0.144016] ** ** [ 0.145013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146016] ************************************************************* [ 0.147734] NET: Registered protocol family 16 [ 0.148628] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.149068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.150064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.151612] cpuidle: using governor menu [ 0.155347] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.160670] PCI: Using configuration type 1 for base access [ 0.164127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.174016] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.175017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.177097] cryptd: max_cpu_qlen set to 1000 [ 0.179233] ACPI: Added _OSI(Module Device) [ 0.180012] ACPI: Added _OSI(Processor Device) [ 0.181015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.182015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.187032] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.193113] ACPI: Interpreter enabled [ 0.194088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.195013] ACPI: Using IOAPIC for interrupt routing [ 0.196194] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.197378] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.215389] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.216939] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.217022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.218082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.220567] acpiphp: Slot [2] registered [ 0.221336] acpiphp: Slot [5] registered [ 0.222233] acpiphp: Slot [6] registered [ 0.223439] acpiphp: Slot [7] registered [ 0.224579] acpiphp: Slot [8] registered [ 0.225190] acpiphp: Slot [9] registered [ 0.226950] acpiphp: Slot [10] registered [ 0.227244] acpiphp: Slot [3] registered [ 0.228144] acpiphp: Slot [4] registered [ 0.229583] acpiphp: Slot [11] registered [ 0.230156] acpiphp: Slot [12] registered [ 0.231119] acpiphp: Slot [13] registered [ 0.232099] acpiphp: Slot [14] registered [ 0.233103] acpiphp: Slot [15] registered [ 0.234165] acpiphp: Slot [16] registered [ 0.235105] acpiphp: Slot [17] registered [ 0.236127] acpiphp: Slot [18] registered [ 0.237134] acpiphp: Slot [19] registered [ 0.238097] acpiphp: Slot [20] registered [ 0.240070] acpiphp: Slot [21] registered [ 0.241239] acpiphp: Slot [22] registered [ 0.242340] acpiphp: Slot [23] registered [ 0.243119] acpiphp: Slot [24] registered [ 0.244955] acpiphp: Slot [25] registered [ 0.245503] acpiphp: Slot [26] registered [ 0.246131] acpiphp: Slot [27] registered [ 0.247117] acpiphp: Slot [28] registered [ 0.248123] acpiphp: Slot [29] registered [ 0.249092] acpiphp: Slot [30] registered [ 0.250104] acpiphp: Slot [31] registered [ 0.251056] PCI host bridge to bus 0000:00 [ 0.252016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.253023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.254024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.255026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.256023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.257030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.258181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.260659] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.263154] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.273922] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.278173] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.279019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.280020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.281020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.283868] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.285340] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.286047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.288155] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.291015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.301017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.304019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.310963] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.313015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.316018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.324018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.333000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.360041] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.387017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.436144] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.488000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.499023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.509022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.535023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.551115] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.578019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.598018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.638135] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.674108] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.704025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.729097] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.746000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.762154] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.775022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.795022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.827022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.856089] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.861148] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.867200] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.871612] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.874308] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.879052] iommu: Default domain type: Passthrough [ 0.883606] SCSI subsystem initialized [ 0.887269] ACPI: bus type USB registered [ 0.891157] usbcore: registered new interface driver usbfs [ 0.899145] usbcore: registered new interface driver hub [ 0.906125] usbcore: registered new device driver usb [ 0.915507] pps_core: LinuxPPS API ver. 1 registered [ 0.921019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.934094] PTP clock support registered [ 0.940355] EDAC MC: Ver: 3.0.0 [ 0.942098] PCI: Using ACPI for IRQ routing [ 0.947415] NetLabel: Initializing [ 0.950018] NetLabel: domain hash size = 128 [ 0.954017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.959590] NetLabel: unlabeled traffic allowed by default [ 0.966056] vgaarb: loaded [ 0.970505] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.976022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.986656] clocksource: Switched to clocksource kvm-clock [ 1.239691] VFS: Disk quotas dquot_6.6.0 [ 1.243658] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.246780] *** VALIDATE ramfs *** [ 1.248764] *** VALIDATE hugetlbfs *** [ 1.250385] pnp: PnP ACPI init [ 1.253230] pnp: PnP ACPI: found 6 devices [ 1.275738] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.279406] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.281936] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.284431] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.287040] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.289873] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.293521] NET: Registered protocol family 2 [ 1.297283] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.303679] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.307524] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.314816] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.320534] TCP: Hash tables configured (established 65536 bind 65536) [ 1.323777] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.327402] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.332545] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.336400] NET: Registered protocol family 1 [ 1.339339] RPC: Registered named UNIX socket transport module. [ 1.341554] RPC: Registered udp transport module. [ 1.347629] RPC: Registered tcp transport module. [ 1.349517] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.352297] NET: Registered protocol family 44 [ 1.354243] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.356508] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.358831] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.362519] PCI: CLS 0 bytes, default 64 [ 1.364604] Unpacking initramfs... [ 4.620775] debug: unmapping init [mem 0xffff9f797cc54000-0xffff9f797ffbffff] [ 4.660937] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.671289] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.696288] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 5.818816] Initialise system trusted keyrings [ 5.821070] Key type blacklist registered [ 5.824040] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.852250] zbud: loaded [ 5.863665] *** VALIDATE nfs *** [ 5.864833] *** VALIDATE nfs4 *** [ 5.872537] pstore: using deflate compression [ 5.883189] Platform Keyring initialized [ 6.140976] NET: Registered protocol family 38 [ 6.143730] Key type asymmetric registered [ 6.145816] Asymmetric key parser 'x509' registered [ 6.148815] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.152281] io scheduler mq-deadline registered [ 6.156210] io scheduler kyber registered [ 6.158822] io scheduler bfq registered [ 6.161387] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.166159] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.169905] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.194807] ACPI: Power Button [PWRF] [ 6.204225] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.216736] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.231834] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.246314] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.283788] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.317483] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.363334] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.379347] Non-volatile memory driver v1.3 [ 6.382280] Linux agpgart interface v0.103 [ 6.445997] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 6.450838] vda: detected capacity change from 0 to 74694656 [ 6.513931] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.522564] vdb: detected capacity change from 0 to 1073741824 [ 6.569718] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.577096] vdc: detected capacity change from 0 to 2621440000 [ 6.625265] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.632296] vdd: detected capacity change from 0 to 2621440000 [ 6.664357] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.671067] vde: detected capacity change from 0 to 4294967296 [ 6.708196] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.713667] vdf: detected capacity change from 0 to 4294967296 [ 6.729608] libphy: Fixed MDIO Bus: probed [ 6.744348] usbcore: registered new interface driver usbserial_generic [ 6.748128] usbserial: USB Serial support registered for generic [ 6.754340] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.762991] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.766689] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.773943] mousedev: PS/2 mouse device common for all mice [ 6.780395] rtc_cmos 00:05: RTC can wake from S4 [ 6.785104] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.787228] rtc_cmos 00:05: registered as rtc0 [ 6.795314] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.799588] intel_pstate: CPU model not supported [ 6.802591] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.812682] hid: raw HID events driver (C) Jiri Kosina [ 6.815308] usbcore: registered new interface driver usbhid [ 6.819097] usbhid: USB HID core driver [ 6.819779] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.823153] drop_monitor: Initializing network drop monitor service [ 6.823384] Initializing XFRM netlink socket [ 6.823869] NET: Registered protocol family 10 [ 6.833849] Segment Routing with IPv6 [ 6.838956] NET: Registered protocol family 17 [ 6.841270] mpls_gso: MPLS GSO support [ 6.847863] RAS: Correctable Errors collector initialized. [ 6.850401] AVX version of gcm_enc/dec engaged. [ 6.852337] AES CTR mode by8 optimization enabled [ 6.964472] sched_clock: Marking stable (6964449135, 0)->(8687543334, -1723094199) [ 6.968278] registered taskstats version 1 [ 6.971293] Loading compiled-in X.509 certificates [ 6.973867] zswap: loaded using pool lzo/zbud [ 7.020673] Key type big_key registered [ 7.038717] Key type encrypted registered [ 7.040916] ima: No TPM chip found, activating TPM-bypass! [ 7.043542] ima: Allocated hash algorithm: sha1 [ 7.045722] ima: No architecture policies found [ 7.047877] evm: Initialising EVM extended attributes: [ 7.050520] evm: security.selinux [ 7.051974] evm: security.ima [ 7.053430] evm: security.capability [ 7.055155] evm: HMAC attrs: 0x1 [ 7.058355] rtc_cmos 00:05: setting system clock to 2026-08-11 02:21:27 UTC (1786414887) [ 7.067324] debug: unmapping init [mem 0xffffffffb9803000-0xffffffffb99fffff] [ 7.071319] debug: unmapping init [mem 0xffffffffb8582000-0xffffffffb8858fff] [ 7.080627] Write protecting the kernel read-only data: 28672k [ 7.097424] debug: unmapping init [mem 0xffffffffb6c03000-0xffffffffb6dfffff] [ 7.101879] debug: unmapping init [mem 0xffffffffb7514000-0xffffffffb75fffff] [ 7.149598] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.179541] systemd[1]: Detected virtualization kvm. [ 7.185937] systemd[1]: Detected architecture x86-64. [ 7.193263] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.247558] systemd[1]: No hostname configured. [ 7.249953] systemd[1]: Set hostname to . [ 7.253629] random: systemd: uninitialized urandom read (16 bytes read) [ 7.256735] systemd[1]: Initializing machine ID from random generator. [ 7.498331] random: ln: uninitialized urandom read (6 bytes read) [ 7.889140] random: systemd: uninitialized urandom read (16 bytes read) [ 7.895617] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 7.906543] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 7.926179] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Journal Service... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.413198] device-mapper: uevent: version 1.0.3 [ 11.420148] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 14.280688] virtio_net virtio0 ens2: renamed from eth0 [ 14.661983] scsi host0: ata_piix [ 14.673270] random: fast init done [ 14.969295] scsi host1: ata_piix [ 14.978170] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 14.984636] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 19.571528] random: crng init done [ 19.574355] random: 7 urandom warning(s) missed due to ratelimiting [ 22.850009] hrtimer: interrupt took 2951440 ns [ 23.067502] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 24.818975] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 28.816680] printk: systemd: 26 output lines suppressed due to ratelimiting [ 30.186663] SELinux: Disabled at runtime. [ 30.354688] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 30.384677] systemd[1]: Detected virtualization kvm. [ 30.394479] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 32.873301] systemd[1]: initrd-switch-root.service: Succeeded. [ 32.887655] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 32.910138] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 32.924276] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 32.952180] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 32.987522] systemd[1]: Starting Journal Service... Starting Journal Service... [ 33.038151] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ 33.725209] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 35.056815] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 36.621716] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 36.737218] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 38.056756] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 38.185251] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 45.123740] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit)[ 46.335683] NFS: Registering the id_resolver key type [ 46.342403] Key type id_resolver registered [ 46.343594] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg442-server login: [ 123.068878] libcfs: loading out-of-tree module taints kernel. [ 123.114818] Key type ._llcrypt registered [ 123.118253] Key type .llcrypt registered [ 123.228552] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 148.097742] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 151.342680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 151.369177] alg: No test for adler32 (adler32-zlib) [ 152.952390] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 153.820293] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 155.559177] Key type lgssc registered [ 158.100585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 179.057479] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 228.723853] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 246.030775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 246.050824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 247.502174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 247.552688] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 247.706901] Lustre: lustre-MDT0000: new disk, initializing [ 247.858509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 247.918456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 253.243358] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 268.430091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 268.536390] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 268.564473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 268.571835] Lustre: Skipped 1 previous similar message [ 268.641865] Lustre: lustre-MDT0001: new disk, initializing [ 268.697990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 268.737442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 268.750454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 274.413043] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.390637] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 291.175045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 291.362456] Lustre: lustre-OST0000: new disk, initializing [ 291.367513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 291.374511] Lustre: 8456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 291.475111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 297.613842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 297.641949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 297.712468] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 299.682372] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 318.148159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 318.280295] Lustre: lustre-OST0001: new disk, initializing [ 318.284038] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 318.288338] Lustre: 9530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 318.369620] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 325.808714] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 326.758725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 326.784696] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 326.921620] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 339.664372] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 347.103377] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 353.612262] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing check_logdir /tmp/testlogs/ [ 360.100745] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing yml_node [ 366.309694] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 370.187529] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 373.588946] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 375.981875] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 10 22:27:34 EDT 2026 [ 397.252255] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 399.023904] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 417.249507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 417.261383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 417.281670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 419.302293] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.305369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 419.344212] Lustre: Skipped 3 previous similar messages [ 421.909573] Lustre: server umount lustre-MDT0000 complete [ 426.095530] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415306 with bad export cookie 2751402526513864609 [ 426.097425] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 426.102763] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 429.539178] LustreError: 8741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 429.543502] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 429.544403] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 429.557494] LustreError: 8741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 429.569450] Lustre: Skipped 3 previous similar messages [ 431.585339] LustreError: 8741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 431.590396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 431.606822] LustreError: 8741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 431.624975] Lustre: Skipped 2 previous similar messages [ 432.484965] Lustre: server umount lustre-MDT0001 complete [ 437.156768] Lustre: server umount lustre-OST0000 complete [ 441.711355] Lustre: server umount lustre-OST0001 complete [ 449.348827] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 460.650563] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 505.096984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 505.542413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 505.590209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 505.675613] Lustre: lustre-MDT0000: new disk, initializing [ 505.816787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.854526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 511.741154] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 523.090530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.345837] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 523.459188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 523.466438] Lustre: Skipped 1 previous similar message [ 523.530701] Lustre: lustre-MDT0001: new disk, initializing [ 523.643863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 523.680486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 523.696328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 530.446949] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 540.501925] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 547.040145] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 555.560455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 556.036511] Lustre: lustre-OST0000: new disk, initializing [ 556.046493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 556.059836] Lustre: 17053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 556.268778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 557.993441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 558.044297] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 558.459283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 565.429593] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 576.270597] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 581.113929] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 581.327625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 586.061755] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 586.345942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 589.288317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.291250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 589.292090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 589.307473] Lustre: Skipped 1 previous similar message [ 589.317770] LustreError: Skipped 1 previous similar message [ 594.861072] Lustre: server umount lustre-OST0000 complete [ 599.524858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.526370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 599.546543] Lustre: Skipped 1 previous similar message [ 599.563060] Lustre: Skipped 4 previous similar messages [ 613.343141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 613.637537] Lustre: server umount lustre-MDT0000 complete [ 614.882883] LustreError: 16315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.910011] LustreError: 16315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 617.680122] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415498 with bad export cookie 2751402526513866142 [ 617.685076] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 617.690768] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 617.930550] Lustre: server umount lustre-MDT0001 complete [ 623.742617] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:31:42 (1786415502) [ 643.511277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.923206] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.992730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 649.189346] LustreError: 19934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 649.300134] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 659.436517] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.475445] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 659.817764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 665.550995] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 675.371443] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 682.109713] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 684.778437] Lustre: Failing over lustre-MDT0000 [ 685.035992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 685.058623] LustreError: 19934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 685.089198] LustreError: 19934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 685.169649] Lustre: server umount lustre-MDT0000 complete [ 701.410668] Lustre: 3650:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786415565/real 1786415565] req@ffff9f79c4be7b80 x1873191930861184/t0(0) o400->MGC192.168.204.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786415581 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 701.489649] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 706.140288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.751946] LustreError: 3646:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f79f7acf480 x1873191930864256/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 711.104347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 711.112784] Lustre: Skipped 1 previous similar message [ 711.206403] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 716.187514] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 716.259801] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 716.262128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 716.289076] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 726.208631] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 731.557244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 731.576228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 731.579039] Lustre: Skipped 2 previous similar messages [ 731.592511] Lustre: Skipped 4 previous similar messages [ 736.737669] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 736.760360] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 736.913813] Lustre: server umount lustre-MDT0000 complete [ 741.229074] LustreError: 19915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415621 with bad export cookie 2751402526513867556 [ 741.236207] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 741.245522] LustreError: 19915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 741.739617] Lustre: server umount lustre-MDT0001 complete [ 750.735507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 755.938568] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 765.083904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 769.834858] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 778.655673] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 785.156909] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 787.412266] Lustre: Failing over lustre-MDT0000 [ 787.633713] Lustre: server umount lustre-MDT0000 complete [ 790.499857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.740696] LustreError: 23270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 800.765114] LustreError: 23270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 805.863350] Lustre: 3648:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786415670/real 1786415670] req@ffff9f79f7e44380 x1873191930891008/t0(0) o400->MGC192.168.204.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786415686 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 805.921916] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 809.374529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.608731] LustreError: 3646:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f79f7e46680 x1873191930893696/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 817.080044] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 821.576375] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 822.259940] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 822.275323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 822.290610] Lustre: Skipped 1 previous similar message [ 822.356750] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 833.125819] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 838.112462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 838.124680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 838.134701] Lustre: Skipped 1 previous similar message [ 843.165988] Lustre: server umount lustre-MDT0000 complete [ 847.201712] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415727 with bad export cookie 2751402526513868494 [ 847.227140] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 847.556911] Lustre: server umount lustre-MDT0001 complete [ 856.894686] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:35:35 (1786415735) [ 873.270986] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 883.402382] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 927.071227] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 939.800724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.188686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 940.254578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 940.389907] Lustre: lustre-MDT0000: new disk, initializing [ 940.528692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 940.540840] Lustre: Skipped 3 previous similar messages [ 940.575134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 945.921572] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 956.998401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.127963] Lustre: 29649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 957.342134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 963.939458] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 976.077990] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 983.231387] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 991.419660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 991.619739] Lustre: 31617:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 993.721708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 993.836431] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 993.959830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 994.015828] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 998.728943] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1010.921319] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1017.754079] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1039.842657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1039.849128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1039.860424] Lustre: Skipped 4 previous similar messages [ 1039.860793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1039.860798] Lustre: Skipped 2 previous similar messages [ 1039.904589] LustreError: Skipped 1 previous similar message [ 1043.499638] Lustre: server umount lustre-OST0000 complete [ 1049.570890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1054.843284] Lustre: server umount lustre-MDT0000 complete [ 1055.213186] LustreError: 29654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1055.264604] LustreError: 29654:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 1058.903072] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415939 with bad export cookie 2751402526513868893 [ 1058.918375] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1058.922836] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1059.361100] Lustre: server umount lustre-MDT0001 complete [ 1070.742142] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1074.222752] Key type lgssc unregistered [ 1074.725768] LNet: 33601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.735581] LNetError: 33601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.748693] LNet: Removed LNI 192.168.204.142@tcp [ 1075.815953] Key type .llcrypt unregistered [ 1075.827701] Key type ._llcrypt unregistered [ 1104.522248] Key type ._llcrypt registered [ 1104.524423] Key type .llcrypt registered [ 1104.632595] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1106.025284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1106.180210] alg: No test for adler32 (adler32-zlib) [ 1107.662806] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 1108.006807] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1109.647212] Key type lgssc registered [ 1111.070215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1119.329514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1119.345600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.831808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1126.270529] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1130.979858] LustreError: 35017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1131.002990] LustreError: 35017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1136.099483] LustreError: 35018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1136.611967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1136.889426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1141.026539] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1149.584144] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1155.390790] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1163.416742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1163.881370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.184784] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1171.465890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1178.660944] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1182.331528] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1200.144151] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:41:18 (1786416078) [ 1202.768915] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1204.822242] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1213.323956] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:41:31 (1786416091) [ 1223.718102] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:41:42 (1786416102) [ 1225.153894] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1235.616424] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1235.620895] Lustre: Skipped 2 previous similar messages [ 1237.822901] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:41:56 (1786416116) [ 1239.848471] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1239.858457] Lustre: Skipped 2 previous similar messages [ 1249.511629] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1249.526602] Lustre: Skipped 4 previous similar messages [ 1267.566604] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:42:26 (1786416146) [ 1270.742881] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1275.399893] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1286.881804] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:42:45 (1786416165) [ 1289.078849] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1289.083320] Lustre: Skipped 6 previous similar messages [ 1304.732171] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:43:03 (1786416183) [ 1325.036376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1325.048622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.073759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1326.053128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1326.055768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1326.070389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.090977] Lustre: Skipped 1 previous similar message [ 1331.175557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1331.183326] Lustre: Skipped 1 previous similar message [ 1336.294307] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1336.313211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1336.335078] Lustre: Skipped 1 previous similar message [ 1336.628557] Lustre: server umount lustre-OST0000 complete [ 1341.410487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.417705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1341.442245] Lustre: Skipped 1 previous similar message [ 1347.013448] Lustre: server umount lustre-MDT0000 complete [ 1351.522996] LustreError: 37938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416231 with bad export cookie 1250804264255066426 [ 1351.534905] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1351.547097] LustreError: 37938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1351.966269] Lustre: server umount lustre-MDT0001 complete [ 1366.699190] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1370.286691] Key type lgssc unregistered [ 1370.633144] LNet: 42888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.644225] LNetError: 42888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1370.660705] LNet: Removed LNI 192.168.204.142@tcp [ 1371.618975] Key type .llcrypt unregistered [ 1371.621103] Key type ._llcrypt unregistered [ 1391.915698] Key type ._llcrypt registered [ 1391.920252] Key type .llcrypt registered [ 1392.185894] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 1408.900666] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1409.917427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1410.155042] alg: No test for adler32 (adler32-zlib) [ 1411.259250] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 1411.459085] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1413.119186] Key type lgssc registered [ 1414.478785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1453.504525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1453.534503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1454.927903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1454.953154] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1455.053122] Lustre: lustre-MDT0000: new disk, initializing [ 1455.139213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.160489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1460.705830] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1472.903227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.072662] Lustre: 46047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1473.147715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1473.154449] Lustre: Skipped 1 previous similar message [ 1473.257379] Lustre: lustre-MDT0001: new disk, initializing [ 1473.383283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1473.435701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1473.450155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1478.612873] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1488.654894] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1495.507429] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1503.232850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1503.581489] Lustre: lustre-OST0000: new disk, initializing [ 1503.585343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1503.597258] Lustre: 48014:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1503.684239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1511.495196] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1513.249275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1513.294322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1513.710267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1524.334779] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1531.047691] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1531.496754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1537.944757] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1538.261950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1544.163396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1544.170310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.189819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1544.677938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.685784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1544.701799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1546.681894] Lustre: server umount lustre-OST0000 complete [ 1554.913078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1554.916098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1554.926770] Lustre: Skipped 1 previous similar message [ 1554.950672] Lustre: Skipped 1 previous similar message [ 1557.065326] Lustre: server umount lustre-MDT0000 complete [ 1561.557621] LustreError: 46039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416441 with bad export cookie 3633600018338765878 [ 1561.564995] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1561.570858] LustreError: 46039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1561.983938] Lustre: server umount lustre-MDT0001 complete [ 1574.006395] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:47:32 (1786416452) [ 1581.309282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1581.781924] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1581.803618] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1581.942232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.491740] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1587.176984] LustreError: 50475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1592.299735] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.379976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1595.662986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1600.852630] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1610.573834] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1617.167619] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1625.264826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1625.755788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.298340] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1644.697423] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1650.196315] LustreError: 53004:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1651.306211] Lustre: Modifying parameter general.timeout=1 in log params [ 1652.632919] Lustre: Modifying parameter general.timeout=2 in log params [ 1655.555669] Lustre: Modifying parameter general.timeout=5 in log params [ 1655.562821] Lustre: Skipped 2 previous similar messages [ 1660.573420] Lustre: Modifying parameter general.timeout=9 in log params [ 1660.582567] Lustre: Skipped 3 previous similar messages [ 1668.818971] Lustre: Modifying parameter general.timeout=16 in log params [ 1668.828628] Lustre: Skipped 6 previous similar messages [ 1685.656119] Lustre: Modifying parameter general.timeout=33 in log params [ 1685.661127] Lustre: Skipped 16 previous similar messages [ 1718.012446] Lustre: Modifying parameter general.timeout=65 in log params [ 1718.022703] Lustre: Skipped 31 previous similar messages [ 1789.113108] Lustre: Disabling parameter general.timeout= in log params [ 1789.121939] Lustre: Skipped 64 previous similar messages [ 1794.260341] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:51:12 (1786416672) [ 1798.157601] Lustre: 59965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1800.624850] Lustre: 60063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1802.899433] Lustre: 60162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1807.212572] Lustre: 60369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1807.229698] Lustre: 60369:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1842.203993] Lustre: 61838:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1842.210970] Lustre: 61838:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1842.228713] LustreError: 61838:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1847.307829] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:52:05 (1786416725) [ 1862.117522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1862.126857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.155360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.211492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.223344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.731298] Lustre: server umount lustre-MDT0000 complete [ 1871.344833] LustreError: 50475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1871.386894] LustreError: 50475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1872.364708] LustreError: 52319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416752 with bad export cookie 3633600018338766760 [ 1872.371590] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1872.380636] LustreError: 52319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1872.733418] Lustre: server umount lustre-MDT0001 complete [ 1887.463694] Lustre: server umount lustre-OST0000 complete [ 1914.614769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1915.093605] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1915.519160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1920.945484] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1929.821121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1929.902357] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1929.924452] Lustre: 64374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1935.387187] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1943.599723] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1949.994623] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1958.020226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1958.225796] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1958.561442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1958.579895] Lustre: Skipped 1 previous similar message [ 1964.786662] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1971.804049] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1973.908374] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1973.917696] Lustre: Skipped 23 previous similar messages [ 1973.965462] Lustre: 66941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1986.532524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.536356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1986.545248] Lustre: Skipped 1 previous similar message [ 1986.548359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1986.553625] LustreError: Skipped 1 previous similar message [ 1986.554835] Lustre: Skipped 2 previous similar messages [ 1988.292773] Lustre: server umount lustre-OST0000 complete [ 1996.769113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2005.983173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2006.200319] Lustre: server umount lustre-MDT0000 complete [ 2007.010699] LustreError: 64384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.032348] LustreError: 64384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2010.330069] LustreError: 64365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416890 with bad export cookie 3633600018338769966 [ 2010.334347] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2010.361842] LustreError: 64365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2010.652302] Lustre: server umount lustre-MDT0001 complete [ 2021.261621] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2024.029547] Key type lgssc unregistered [ 2024.263498] LNet: 68161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2024.277062] LNetError: 68161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2024.290241] LNet: Removed LNI 192.168.204.142@tcp [ 2025.066546] Key type .llcrypt unregistered [ 2025.068591] Key type ._llcrypt unregistered [ 2049.580666] Key type ._llcrypt registered [ 2049.583548] Key type .llcrypt registered [ 2049.702304] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2050.619739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2050.714516] alg: No test for adler32 (adler32-zlib) [ 2051.763842] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 2051.951656] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2053.615218] Key type lgssc registered [ 2055.061437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2063.549970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2063.574162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.017825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.563617] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2074.599225] LustreError: 69578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.628900] LustreError: 69578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2079.533918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.718717] LustreError: 69579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.361574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2086.188575] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2094.202217] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2101.183482] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2109.276798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2109.821128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.833248] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2125.480341] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2133.480945] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2145.765920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.768055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2145.784822] Lustre: Skipped 2 previous similar messages [ 2145.816271] Lustre: Skipped 2 previous similar messages [ 2148.219397] Lustre: server umount lustre-MDT0000 complete [ 2150.888299] LustreError: 70512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.911544] LustreError: 70512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2152.472249] LustreError: 69559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417032 with bad export cookie 6354882228887650987 [ 2152.476842] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2152.491549] LustreError: 69559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2153.053483] Lustre: server umount lustre-MDT0001 complete [ 2167.417514] Lustre: server umount lustre-OST0000 complete [ 2200.350926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.751732] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2200.786504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2201.133608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.135492] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2217.433176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.561668] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2217.599864] Lustre: 74451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2218.205570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2224.259596] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2231.970725] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2237.090454] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2244.284482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2244.509303] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2244.971228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.674588] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2261.253651] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2263.122950] Lustre: 77027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2265.574776] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2271.721776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2271.734793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.738019] LustreError: Skipped 1 previous similar message [ 2271.749731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2271.764530] Lustre: Skipped 1 previous similar message [ 2276.418879] Lustre: server umount lustre-OST0000 complete [ 2281.954419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.957967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2281.965590] Lustre: Skipped 1 previous similar message [ 2281.974579] Lustre: Skipped 1 previous similar message [ 2286.774456] Lustre: server umount lustre-MDT0000 complete [ 2287.080209] LustreError: 74461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.109957] LustreError: 74461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2291.525379] LustreError: 75197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417171 with bad export cookie 6354882228887651820 [ 2291.541397] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2291.991384] Lustre: server umount lustre-MDT0001 complete [ 2305.275881] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2307.967197] Key type lgssc unregistered [ 2308.330410] LNet: 78239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2308.345579] LNetError: 78239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2308.365883] LNet: Removed LNI 192.168.204.142@tcp [ 2309.102153] Key type .llcrypt unregistered [ 2309.105301] Key type ._llcrypt unregistered [ 2338.168497] Key type ._llcrypt registered [ 2338.173316] Key type .llcrypt registered [ 2338.319226] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2339.829659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2339.926694] alg: No test for adler32 (adler32-zlib) [ 2341.072236] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 2341.479108] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2343.303228] Key type lgssc registered [ 2344.785842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2353.780928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2353.811519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2355.378980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2360.687216] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2364.918808] LustreError: 79653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.948287] LustreError: 79653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2370.026304] LustreError: 79652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.972770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.416475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2376.664890] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2385.142618] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2391.380450] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2399.155953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2399.750644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2406.300293] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2413.990300] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2421.137432] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2421.911836] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2423.929992] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2423.937838] Lustre: Skipped 1 previous similar message [ 2426.037355] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2426.042809] Lustre: Skipped 1 previous similar message [ 2430.282629] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2430.289344] Lustre: Skipped 3 previous similar messages [ 2438.405789] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2438.418968] Lustre: Skipped 9 previous similar messages [ 2454.875625] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2454.880887] Lustre: Skipped 17 previous similar messages [ 2487.372360] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2487.379580] Lustre: Skipped 34 previous similar messages [ 2522.882089] Lustre: 87769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2526.837750] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 23:03:25 (1786417405) [ 2528.699566] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2530.671935] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 23:03:29 (1786417409) [ 2534.370781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2534.381048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.400367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2537.444810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2537.450278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.499328] Lustre: server umount lustre-OST0000 complete [ 2547.681271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.687861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2547.704436] Lustre: Skipped 1 previous similar message [ 2547.734627] Lustre: Skipped 2 previous similar messages [ 2551.597108] Lustre: server umount lustre-MDT0000 complete [ 2552.812454] LustreError: 79652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.877249] LustreError: 79652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2555.765926] LustreError: 79634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417436 with bad export cookie 13088679566324262393 [ 2555.781388] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2556.114870] Lustre: server umount lustre-MDT0001 complete [ 2570.491205] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2573.594645] Key type lgssc unregistered [ 2573.868902] LNet: 89022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2573.874179] LNetError: 89022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2574.951055] LNet: Removed LNI 192.168.204.142@tcp [ 2575.951802] Key type .llcrypt unregistered [ 2575.954749] Key type ._llcrypt unregistered [ 2581.337950] Key type ._llcrypt registered [ 2581.340840] Key type .llcrypt registered [ 2581.535616] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2588.860375] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules [ 2591.216719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2591.259922] alg: No test for adler32 (adler32-zlib) [ 2592.589571] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 2592.963960] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2594.831052] Key type lgssc registered [ 2596.868845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2599.666753] LustreError: 90307:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2618.450259] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2621.553523] LustreError: 90307:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2631.427546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2631.441554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.092370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2638.672612] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2643.429669] LustreError: 91301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.454769] LustreError: 91301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2647.231114] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 23:05:26 (1786417526) [ 2648.547189] LustreError: 91302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.005742] LustreError: 91334:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2652.027943] LustreError: 91334:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2652.045271] LustreError: 91334:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2652.406622] Lustre: server umount lustre-MDT0000 complete [ 2667.314948] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2670.605857] Key type lgssc unregistered [ 2670.912282] LNet: 92831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2670.923494] LNetError: 92831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2671.973756] LNet: Removed LNI 192.168.204.142@tcp [ 2672.815200] Key type .llcrypt unregistered [ 2672.817333] Key type ._llcrypt unregistered [ 2695.717416] Key type ._llcrypt registered [ 2695.720408] Key type .llcrypt registered [ 2695.861852] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2697.481658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2697.563165] alg: No test for adler32 (adler32-zlib) [ 2698.600390] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 2698.811464] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2700.455178] Key type lgssc registered [ 2701.798609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2710.156214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2710.186994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2711.878485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.688483] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2722.275062] LustreError: 94052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.297744] LustreError: 94052:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2725.910736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.577199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2732.598495] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2740.752621] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2748.586782] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2756.386897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.100839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2764.090326] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2771.772606] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2819.378961] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 23:08:17 (1786417697) [ 2821.381955] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2823.645551] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 23:08:22 (1786417702) [ 2827.758665] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2827.788194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2832.868146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2832.874622] Lustre: Skipped 4 previous similar messages [ 2834.342167] Lustre: server umount lustre-MDT0000 complete [ 2837.987860] LustreError: 94783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2838.008917] LustreError: 94783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2838.377940] LustreError: 94035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417718 with bad export cookie 4846845688508894749 [ 2838.393587] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2838.403859] LustreError: 94035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2838.722632] Lustre: server umount lustre-MDT0001 complete [ 2854.034107] Lustre: server umount lustre-OST0000 complete [ 2866.747156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.075147] LustreError: 98326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.227527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.112383] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2872.294964] LustreError: 98327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.412769] LustreError: 98326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2881.774141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.484910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2888.839730] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2897.833914] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2904.940677] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2920.207429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2920.367634] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2920.374222] LustreError: 98315:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2920.402913] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2920.410757] LustreError: 100412:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2920.417938] LustreError: 100412:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2920.421560] LustreError: 100412:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2920.505980] Lustre: server umount lustre-OST0000 complete [ 2920.513273] LustreError: 100412:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2930.234810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2930.513876] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2930.535896] LustreError: 98316:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2930.597929] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2930.617488] LustreError: 100878:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2930.624786] LustreError: 100878:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2930.629394] LustreError: 100878:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2930.758655] Lustre: server umount lustre-OST0000 complete [ 2930.768679] LustreError: 100878:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2942.404810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2942.549312] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2942.555808] Lustre: Found index 0 for lustre-OST0000, updating log [ 2942.566971] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2942.621108] Lustre: lustre-OST0000: new disk, initializing [ 2942.625560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2942.635339] Lustre: 101412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2942.836276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2950.259867] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2954.450324] Lustre: Failing over lustre-OST0000 [ 2954.715682] Lustre: server umount lustre-OST0000 complete [ 2962.400275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2962.409568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2962.421057] Lustre: Skipped 3 previous similar messages [ 2967.525064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2967.530021] Lustre: Skipped 2 previous similar messages [ 2972.575184] Lustre: 93365:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786417837/real 1786417837] req@000000006dd9a430 x1873194599420416/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786417853 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2972.611418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.631880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.175174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2974.327062] Lustre: server umount lustre-MDT0000 complete [ 2977.763720] LustreError: 98322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.764894] Lustre: 93363:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786417842/real 1786417842] req@00000000b93e2d10 x1873194599422336/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786417858 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2977.799610] LustreError: 98322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2977.861759] Lustre: 93363:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2978.638040] LustreError: 98306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417859 with bad export cookie 4846845688508897493 [ 2978.654195] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2979.076341] Lustre: server umount lustre-MDT0001 complete [ 2989.877881] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 23:11:08 (1786417868) [ 3008.771958] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3023.775532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.261064] LustreError: 103762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.462971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3030.510206] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3040.221276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.325579] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3049.137364] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3056.562990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3057.161688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3057.168609] Lustre: Skipped 1 previous similar message [ 3057.180744] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3065.401875] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3065.433966] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3065.442971] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3065.445218] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3076.078546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3076.238808] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3076.405971] Lustre: lustre-OST0001: new disk, initializing [ 3076.415103] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3076.432967] Lustre: 106080:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3084.987263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3085.016200] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3085.303696] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3086.678834] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3095.409718] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3102.078193] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3107.300580] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3107.306747] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.319665] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3107.325382] Lustre: Skipped 1 previous similar message [ 3112.865083] Lustre: server umount lustre-MDT0001 complete [ 3115.518914] LustreError: 103759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3115.554290] LustreError: 103759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3125.804220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3125.988857] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3126.009447] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3126.075871] Lustre: 103751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3126.259412] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3126.281162] Lustre: Skipped 4 previous similar messages [ 3127.092389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3127.104231] Lustre: Skipped 1 previous similar message [ 3128.029215] LustreError: 107639:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3128.047452] LustreError: 107639:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3128.055058] LustreError: 103754:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3128.165657] LustreError: 107657:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.142@tcp: cfg command failed: rc = -17 [ 3128.202571] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3128.202571] [ 3132.425600] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3133.703615] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3146.995092] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 23:13:45 (1786418025) [ 3174.073393] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3198.734304] Lustre: 109721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3226.422333] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3231.084617] Lustre: 110856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3326.437867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3326.449288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3326.465358] Lustre: Skipped 2 previous similar messages [ 3326.474353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3331.326881] Lustre: server umount lustre-MDT0000 complete [ 3332.083150] LustreError: 103759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3332.109705] LustreError: 103759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3336.086466] LustreError: 106079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418216 with bad export cookie 4846845688508898501 [ 3336.101989] LustreError: 106079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3336.102086] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3336.590724] Lustre: server umount lustre-MDT0001 complete [ 3353.125891] Lustre: server umount lustre-OST0000 complete [ 3353.442994] Lustre: 93363:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786418217/real 1786418217] req@ffff9f79c83c9180 x1873194599698560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786418233 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3353.469144] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.482614] Lustre: Skipped 3 previous similar messages [ 3357.262503] Lustre: server umount lustre-OST0001 complete [ 3364.452611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3374.872348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3384.276408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3397.975284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3425.583793] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3438.649959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.704487] Lustre: lustre-MDT0000: reset Object Index mappings [ 3439.222314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3444.855917] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3455.380318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.396668] Lustre: lustre-MDT0001: reset Object Index mappings [ 3462.852578] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3467.085852] Lustre: 115962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3476.299417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3481.403360] LustreError: 116317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3481.430336] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3481.446389] LustreError: 116317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3484.554199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3485.450731] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3496.651495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3502.628233] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3502.647967] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3505.002740] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3515.440534] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3521.361287] Lustre: 117839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3577.728895] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 23:20:56 (1786418456) [ 3604.452069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3604.463892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3604.489535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3604.497169] Lustre: Skipped 3 previous similar messages [ 3609.367997] Lustre: server umount lustre-MDT0000 complete [ 3613.264933] LustreError: 114791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418493 with bad export cookie 4846845688508945373 [ 3613.282954] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3613.290216] LustreError: 114791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3619.812948] Lustre: server umount lustre-MDT0001 complete [ 3638.239343] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3638.372830] Lustre: server umount lustre-OST0000 complete [ 3649.409508] Lustre: server umount lustre-OST0001 complete [ 3659.508717] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 3676.793417] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3722.659143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.993244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3723.036085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3723.131733] Lustre: lustre-MDT0000: new disk, initializing [ 3723.336351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3723.345480] Lustre: Skipped 3 previous similar messages [ 3723.391937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3730.015563] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3738.275974] Lustre: server umount lustre-MDT0000 complete [ 3751.084356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3751.524320] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3751.539557] Lustre: 122133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3751.553612] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3756.816477] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3763.393631] Lustre: server umount lustre-MDT0000 complete [ 3776.458679] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 23:24:14 (1786418654) [ 3779.244721] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3782.216166] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 23:24:20 (1786418660) [ 3807.234849] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3820.120091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3825.958431] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3837.514401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.686915] Lustre: 124204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3837.776459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3837.780267] Lustre: Skipped 1 previous similar message [ 3837.843478] Lustre: lustre-MDT0001: new disk, initializing [ 3837.983932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3838.002566] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3843.531725] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3848.962355] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3858.185456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3858.440771] Lustre: lustre-OST0000: new disk, initializing [ 3858.444839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3858.450618] Lustre: 125786:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3859.749950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3859.785505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3860.116583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3866.576411] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3879.590837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3879.921647] Lustre: lustre-OST0001: new disk, initializing [ 3879.932932] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3879.957693] Lustre: 126657:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3881.504830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3881.527879] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3881.909949] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3888.975747] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3901.364836] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3906.731853] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4334.062286] Lustre: 133079:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4334.073181] Lustre: 133079:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4334.077726] Lustre: 133079:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4334.084924] Lustre: 133079:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4334.090118] Lustre: 133079:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4334.095154] Lustre: 133079:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4633.802941] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:38:32 (1786419512) [ 4650.979501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4651.000750] Lustre: Skipped 7 previous similar messages [ 4651.002182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4651.026971] Lustre: Skipped 10 previous similar messages [ 4654.837139] Lustre: server umount lustre-MDT0000 complete [ 4656.105748] LustreError: 127462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4656.126799] LustreError: 127462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 4658.677199] LustreError: 124945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419539 with bad export cookie 4846845688509014050 [ 4658.697718] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4658.699488] LustreError: 124945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4661.225633] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4661.242387] Lustre: Skipped 2 previous similar messages [ 4665.222398] Lustre: server umount lustre-MDT0001 complete [ 4684.255086] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4684.370407] Lustre: server umount lustre-OST0000 complete [ 4694.434860] Lustre: server umount lustre-OST0001 complete [ 4702.358465] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 4715.356240] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4756.849174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4757.152179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4757.184414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4757.337436] Lustre: lustre-MDT0000: new disk, initializing [ 4757.457834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4757.473411] Lustre: Skipped 5 previous similar messages [ 4757.522700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4763.314098] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4774.799221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4774.903443] Lustre: 136144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4774.958717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4774.975389] Lustre: Skipped 1 previous similar message [ 4775.038656] Lustre: lustre-MDT0001: new disk, initializing [ 4775.153838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4775.164252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4779.497347] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4789.586860] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4795.393279] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4803.337944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4803.808342] Lustre: lustre-OST0000: new disk, initializing [ 4803.823525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4803.830725] Lustre: 138112:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4805.557768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4805.573498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4805.821451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4810.825978] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4821.803373] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4833.476379] Lustre: lustre-MDD0000: changelog on [ 4836.720354] Lustre: lustre-MDD0001: changelog on [ 6749.411758] Lustre: lustre-MDD0001: changelog off [ 6752.245951] Lustre: 136151:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6753.277776] Lustre: lustre-MDD0000: changelog off [ 6761.440450] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6761.450031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6761.456512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6761.467587] Lustre: Skipped 1 previous similar message [ 6761.490365] Lustre: Skipped 5 previous similar messages [ 6764.533775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6764.550218] Lustre: Skipped 1 previous similar message [ 6769.651383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6769.660884] Lustre: Skipped 1 previous similar message [ 6774.242279] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6774.441171] Lustre: server umount lustre-OST0000 complete [ 6779.875317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6779.887759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6779.898946] Lustre: Skipped 1 previous similar message [ 6785.974805] Lustre: server umount lustre-MDT0000 complete [ 6790.114839] LustreError: 136155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6790.131546] LustreError: 136155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6790.566170] LustreError: 136136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421671 with bad export cookie 4846845688509842479 [ 6790.578416] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6790.589775] LustreError: 136136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6790.907388] Lustre: server umount lustre-MDT0001 complete [ 6804.549547] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6807.619571] Key type lgssc unregistered [ 6807.929604] LNet: 147269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6807.937277] LNetError: 147269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6807.955069] LNet: Removed LNI 192.168.204.142@tcp [ 6808.908184] Key type .llcrypt unregistered [ 6808.911855] Key type ._llcrypt unregistered [ 6834.272184] Key type ._llcrypt registered [ 6834.274343] Key type .llcrypt registered [ 6834.374399] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 6852.024453] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6853.295834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6853.392927] alg: No test for adler32 (adler32-zlib) [ 6854.861926] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 6855.185032] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6856.887171] Key type lgssc registered [ 6858.324567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6900.392657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6900.457652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6901.802430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6901.842105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6901.944705] Lustre: lustre-MDT0000: new disk, initializing [ 6902.056383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6902.084282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6907.846680] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6921.504306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6921.837286] Lustre: 150474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6921.932267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6921.938209] Lustre: Skipped 1 previous similar message [ 6922.008170] Lustre: lustre-MDT0001: new disk, initializing [ 6922.170094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6922.273117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6922.298177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6927.549645] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6938.448922] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6945.413359] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6953.250559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6953.540208] Lustre: lustre-OST0000: new disk, initializing [ 6953.543763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6953.549568] Lustre: 152439:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6953.653720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6959.101292] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6961.838499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6961.865620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6962.074635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6969.597663] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6975.153148] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_module obdecho/obdecho [ 6977.250735] LustreError: 153441:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6977.256743] LustreError: 153441:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6979.675622] LustreError: 153538:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6992.869716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6992.879131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6992.893748] Lustre: Skipped 2 previous similar messages [ 6992.894672] Lustre: Skipped 1 previous similar message [ 6997.985926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6998.006833] Lustre: Skipped 3 previous similar messages [ 6998.844892] Lustre: server umount lustre-MDT0000 complete [ 7003.807115] LustreError: 150463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421884 with bad export cookie 1908874578025273225 [ 7003.809478] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7003.820178] LustreError: 150463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7004.102904] Lustre: server umount lustre-MDT0001 complete [ 7017.954337] Lustre: server umount lustre-OST0000 complete [ 7045.218642] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 7058.081584] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7101.816527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7102.074615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7102.096538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7102.144567] Lustre: lustre-MDT0000: new disk, initializing [ 7102.200402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7102.216696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7107.125751] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7118.792859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7118.920311] Lustre: 157263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7118.976151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7118.979376] Lustre: Skipped 1 previous similar message [ 7119.064684] Lustre: lustre-MDT0001: new disk, initializing [ 7119.185647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7119.227467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7119.251212] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7124.641345] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7135.894536] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7143.447804] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7152.127186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7152.392750] Lustre: lustre-OST0000: new disk, initializing [ 7152.395799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7152.404704] Lustre: 159230:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7152.497133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7159.995031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7160.025958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7160.220183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7160.968638] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7171.424526] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7176.577596] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7176.815806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7182.138674] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7182.428932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7185.899700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7185.901684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7185.905994] LustreError: Skipped 1 previous similar message [ 7185.928783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7191.026262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7191.043709] Lustre: Skipped 2 previous similar messages [ 7196.135836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7196.145855] Lustre: Skipped 1 previous similar message [ 7199.202035] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7199.499984] Lustre: server umount lustre-OST0000 complete [ 7206.373685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7206.377803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7206.394041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7206.405814] Lustre: Skipped 2 previous similar messages [ 7206.419418] Lustre: Skipped 1 previous similar message [ 7209.687164] Lustre: server umount lustre-MDT0000 complete [ 7211.489628] LustreError: 157268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7211.509521] LustreError: 157268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7213.504074] LustreError: 159240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422093 with bad export cookie 1908874578025274184 [ 7213.506911] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7213.513864] LustreError: 159240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7213.934129] Lustre: server umount lustre-MDT0001 complete [ 7218.540944] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 00:21:36 (1786422096) [ 7220.618301] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7222.742467] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 00:21:41 (1786422101) [ 7232.389477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7232.756622] LustreError: 161644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7232.837848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7238.115515] LustreError: 161645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7238.489296] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7243.241839] LustreError: 161644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7248.158645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7254.359240] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7261.935790] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7268.010288] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7274.426076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7274.733732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7274.742242] Lustre: Skipped 1 previous similar message [ 7281.013202] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7289.605865] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7300.624259] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [ 7301.526768] Lustre: server umount lustre-MDT0000 complete [ 7303.673890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7303.679115] LustreError: 164135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7303.709829] Lustre: Skipped 2 previous similar messages [ 7303.723697] LustreError: 164135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7305.336690] LustreError: 161626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422185 with bad export cookie 1908874578025275066 [ 7305.342719] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7305.349242] LustreError: 161626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7308.771890] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.936456] Lustre: server umount lustre-MDT0001 complete [ 7320.235678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7336.728156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7337.028556] LustreError: 165488:0:(llog.c:1655:llog_backup()) MGC192.168.204.142@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7337.039967] Lustre: 165488:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7340.577893] LustreError: 165493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7340.606144] LustreError: 165493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7356.911477] LustreError: 165495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7356.931784] LustreError: 165495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7391.202265] LustreError: 165494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7391.224295] LustreError: 165494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 7397.855380] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7397.862353] Lustre: 165488:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.142@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7438.847244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7443.962704] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7445.497321] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7455.214975] LustreError: 163475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7455.229815] LustreError: 163475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 7455.562785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7465.460740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7474.656574] LustreError: 148694:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ad3d4353 x1873198957338624/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7479.509671] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7480.318988] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7486.857670] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7489.070509] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7494.809884] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7496.710179] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7516.130944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7516.140902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7516.162665] Lustre: Skipped 2 previous similar messages [ 7516.190397] Lustre: Skipped 6 previous similar messages [ 7521.337897] Lustre: server umount lustre-MDT0000 complete [ 7525.605205] LustreError: 162379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422406 with bad export cookie 1908874578025281555 [ 7525.610636] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7525.618767] LustreError: 162379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7526.106696] Lustre: server umount lustre-MDT0001 complete [ 7540.637718] Lustre: server umount lustre-OST0000 complete [ 7562.879394] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 7575.251441] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7614.473916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7614.750389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7614.778851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7614.840540] Lustre: lustre-MDT0000: new disk, initializing [ 7614.903650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7614.910339] Lustre: Skipped 1 previous similar message [ 7614.932848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7620.433572] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7632.334913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7632.618749] Lustre: 170598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7633.176215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7638.280966] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7649.330681] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7655.837927] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7662.524155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7662.944600] Lustre: lustre-OST0000: new disk, initializing [ 7662.951080] Lustre: Skipped 1 previous similar message [ 7662.961912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7662.973599] Lustre: Skipped 2 previous similar messages [ 7662.986445] Lustre: 172568:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7669.881800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7669.893024] Lustre: Skipped 1 previous similar message [ 7669.900664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7670.107594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7670.714797] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7681.677358] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7686.908724] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7687.136491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7691.750162] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7691.941860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7695.844093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7695.861782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7695.865376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7695.882866] Lustre: Skipped 1 previous similar message [ 7708.127132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7708.310724] Lustre: server umount lustre-OST0000 complete [ 7714.785234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7718.479578] Lustre: server umount lustre-MDT0000 complete [ 7721.452437] LustreError: 172783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7721.465224] LustreError: 172783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 7722.464378] LustreError: 170592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422602 with bad export cookie 1908874578025298348 [ 7722.476204] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7722.479654] LustreError: 170592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7722.787626] Lustre: server umount lustre-MDT0001 complete [ 7733.712306] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:30:12 (1786422612) [ 7740.915023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7746.756198] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7758.748913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7764.686444] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7773.621646] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7780.605665] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7787.986965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7795.315994] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7803.354974] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7828.066609] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:31:46 (1786422706) [ 7835.620779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7835.639783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7835.663108] Lustre: Skipped 3 previous similar messages [ 7835.674630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7835.684859] Lustre: Skipped 7 previous similar messages [ 7839.439614] Lustre: server umount lustre-OST0000 complete [ 7849.901766] Lustre: server umount lustre-MDT0000 complete [ 7854.416386] LustreError: 177985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422734 with bad export cookie 1908874578025299237 [ 7854.424493] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7854.437035] LustreError: 177985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7854.919780] Lustre: server umount lustre-MDT0001 complete [ 7867.178937] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 7870.453547] Key type lgssc unregistered [ 7870.779159] LNet: 178742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7870.789461] LNetError: 178742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7870.808874] LNet: Removed LNI 192.168.204.142@tcp [ 7871.969133] Key type .llcrypt unregistered [ 7871.974747] Key type ._llcrypt unregistered [ 7896.082666] Key type ._llcrypt registered [ 7896.085946] Key type .llcrypt registered [ 7896.281912] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7898.070393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7898.423673] alg: No test for adler32 (adler32-zlib) [ 7899.997754] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 7900.255892] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 7902.127454] Key type lgssc registered [ 7903.568421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7912.612711] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7912.632600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7928.159528] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7933.279466] LustreError: 179984:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 7938.399621] LustreError: 179984:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7938.412768] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7938.418888] LustreError: MGC192.168.204.142@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7938.427151] LustreError: 179984:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7938.432516] LustreError: 179984:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7938.442699] LustreError: 179984:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7938.485416] Lustre: server umount lustre-OST0000 complete [ 7938.488500] LustreError: 179984:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7948.675616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7950.250047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7956.256695] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7960.548964] LustreError: 180484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7960.577623] LustreError: 180484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7965.665245] LustreError: 180483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7967.443065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7967.905560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7972.998701] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7982.096267] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7988.927888] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7997.334855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7997.510547] LustreError: 182300:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7997.520394] Lustre: 182300:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.142@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7997.693359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8004.680822] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8015.752802] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8023.522943] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8023.525072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8023.541759] Lustre: Skipped 2 previous similar messages [ 8023.549201] Lustre: Skipped 1 previous similar message [ 8025.427520] Lustre: server umount lustre-MDT0000 complete [ 8028.645383] LustreError: 181211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8028.664457] LustreError: 181211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8029.895593] LustreError: 180465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422910 with bad export cookie 6740218490265375993 [ 8029.903520] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8029.906865] LustreError: 180465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8030.198591] Lustre: server umount lustre-MDT0001 complete [ 8045.134757] Lustre: server umount lustre-OST0000 complete [ 8057.142517] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8073.453488] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8120.245814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8120.822180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8120.845034] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8120.986514] Lustre: lustre-MDT0000: new disk, initializing [ 8121.150084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8121.179223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8126.299766] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8139.627323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8139.788576] Lustre: 185851:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8139.849283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8139.853350] Lustre: Skipped 1 previous similar message [ 8139.938744] Lustre: lustre-MDT0001: new disk, initializing [ 8140.068794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8140.108385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8140.132705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8145.601237] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8157.271136] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8162.720934] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8170.834987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8171.250663] Lustre: lustre-OST0000: new disk, initializing [ 8171.255550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8171.267494] Lustre: 187824:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8171.390264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8179.528873] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8180.940944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8180.953590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8181.070210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8189.090502] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8194.816782] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8195.067557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8201.886354] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8202.401958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8206.822292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8206.824193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8206.829408] LustreError: Skipped 1 previous similar message [ 8206.830191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8206.830195] Lustre: Skipped 1 previous similar message [ 8206.848711] Lustre: Skipped 1 previous similar message [ 8211.314396] Lustre: server umount lustre-OST0000 complete [ 8217.059099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8217.067350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8217.084727] Lustre: Skipped 1 previous similar message [ 8217.090760] Lustre: Skipped 2 previous similar messages [ 8222.179764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8222.198334] Lustre: Skipped 1 previous similar message [ 8222.748767] Lustre: server umount lustre-MDT0000 complete [ 8227.138446] LustreError: 185845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423107 with bad export cookie 6740218490265376707 [ 8227.165946] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8227.168674] LustreError: 185845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8227.836447] Lustre: server umount lustre-MDT0001 complete [ 8239.693243] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:38:37 (1786423117) [ 8266.370537] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 8270.130471] Key type lgssc unregistered [ 8270.478835] LNet: 190773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8270.494557] LNetError: 190773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8270.522495] LNet: Removed LNI 192.168.204.142@tcp [ 8271.936818] Key type .llcrypt unregistered [ 8271.939330] Key type ._llcrypt unregistered [ 8294.483202] Key type ._llcrypt registered [ 8294.487355] Key type .llcrypt registered [ 8294.700794] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8296.066707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8296.099473] alg: No test for adler32 (adler32-zlib) [ 8297.510376] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 8297.733957] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 8299.383579] Key type lgssc registered [ 8301.178811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8310.470469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8310.486333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8311.897447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8318.629291] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8322.017134] LustreError: 191993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8322.033074] LustreError: 191993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8327.149666] LustreError: 191992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8329.158935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8329.486590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8334.946651] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8343.333807] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8350.633993] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8360.814419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8360.986282] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8360.988834] LustreError: 193856:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8360.996134] Lustre: 193856:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.142@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8361.107728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8368.684816] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8377.953495] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8380.906817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8380.919664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8380.951150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8385.001253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8385.002300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8390.114711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8390.126340] Lustre: Skipped 4 previous similar messages [ 8395.232358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8395.236258] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 8395.242399] Lustre: Skipped 2 previous similar messages [ 8395.500103] Lustre: server umount lustre-MDT0000 complete [ 8400.367676] LustreError: 191992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8400.436464] LustreError: 191992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8400.837770] LustreError: 191973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423281 with bad export cookie 2642256282671768396 [ 8400.845854] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8400.850798] LustreError: 191973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8401.153835] Lustre: server umount lustre-MDT0001 complete [ 8416.218971] Lustre: server umount lustre-OST0000 complete [ 8428.371194] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8445.585271] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8496.008233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8496.171807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8496.193701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8496.254934] Lustre: lustre-MDT0000: new disk, initializing [ 8496.351961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8496.378956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8501.697585] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8513.104372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8513.318296] Lustre: 197407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8513.394446] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8513.396737] Lustre: Skipped 1 previous similar message [ 8513.402871] LustreError: 198191:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8513.416756] LustreError: 198191:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8513.426844] Lustre: 198191:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8513.460044] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8513.464959] Lustre: Skipped 1 previous similar message [ 8513.542646] Lustre: lustre-MDT0001: new disk, initializing [ 8513.716766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8513.787621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8513.815732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8518.668475] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8530.829702] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8539.030824] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8546.593765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8546.869313] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8546.871855] Lustre: Skipped 1 previous similar message [ 8546.874340] LustreError: 199352:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8546.883960] LustreError: 199352:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8546.891700] Lustre: 199352:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8546.907231] Lustre: 199352:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8546.928915] Lustre: lustre-OST0000: new disk, initializing [ 8546.932116] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8546.937195] Lustre: 199377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8547.030090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8553.127221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8553.160237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8553.369738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8554.540431] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8566.376867] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8574.054501] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8574.422480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8580.137486] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8580.337912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8584.171757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8584.174303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8584.189984] Lustre: Skipped 1 previous similar message [ 8584.195183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8584.202241] LustreError: Skipped 1 previous similar message [ 8584.235404] Lustre: Skipped 1 previous similar message [ 8589.223659] Lustre: server umount lustre-OST0000 complete [ 8595.433112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8595.439402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8595.449447] Lustre: Skipped 1 previous similar message [ 8595.453653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8599.526072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8601.678221] Lustre: server umount lustre-MDT0000 complete [ 8603.682238] LustreError: 197420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8603.701075] LustreError: 197420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8606.076278] LustreError: 197402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423486 with bad export cookie 2642256282671769110 [ 8606.083546] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8606.096853] LustreError: 197402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8606.584424] Lustre: server umount lustre-MDT0001 complete [ 8619.419672] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:44:58 (1786423498) [ 8644.242405] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8657.947530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8658.299920] LustreError: 202535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8658.390880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8664.500321] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8667.643285] LustreError: 202536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8667.688472] LustreError: 202536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8676.587131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8682.976707] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8687.202075] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8695.788133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8696.198419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8696.206887] Lustre: Skipped 1 previous similar message [ 8705.197439] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8716.443105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8716.760723] Lustre: lustre-OST0001: new disk, initializing [ 8716.766750] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8716.777628] Lustre: 204844:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8724.088992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8724.107679] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8724.281788] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8725.225559] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8739.243668] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8748.771414] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8751.579765] Lustre: DEBUG MARKER: ADD OST3 [ 8759.714403] loop: module loaded [ 8759.855156] loop0: detected capacity change from 0 to 204800000 [ 8760.232259] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8760.245869] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8760.257605] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8768.521779] loop0: detected capacity change from 0 to 204800000 [ 8770.149258] Lustre: DEBUG MARKER: STOP OST3 [ 8779.129145] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8779.682854] Lustre: lustre-OST0003: new disk, initializing [ 8779.687209] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8779.695533] Lustre: 206887:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8780.054543] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8780.079735] Lustre: Skipped 1 previous similar message [ 8781.582121] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8781.832435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8781.854560] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8781.912903] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8783.649901] LustreError: 206969:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8785.012370] Lustre: Failing over lustre-OST0003 [ 8785.166715] Lustre: server umount lustre-OST0003 complete [ 8786.482806] LustreError: 204022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8786.517423] LustreError: 204022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8786.937275] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8786.940751] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8786.960168] Lustre: Skipped 1 previous similar message [ 8792.959370] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8806.505976] Lustre: DEBUG MARKER: START OST3 again [ 8816.850470] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8817.131817] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8817.144192] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8819.104167] Lustre: lustre-OST0003: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 8819.115964] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8825.403868] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8851.041986] Lustre: Failing over lustre-OST0003 [ 8851.244170] Lustre: server umount lustre-OST0003 complete [ 8852.978886] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8853.000967] LustreError: 204022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8853.019346] LustreError: 204022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8873.479871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8873.490603] Lustre: Skipped 4 previous similar messages [ 8886.239572] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8886.447710] Lustre: server umount lustre-MDT0000 complete [ 8890.714049] LustreError: 202515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423771 with bad export cookie 2642256282671769992 [ 8890.715340] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8890.726297] LustreError: 202515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8893.923515] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8893.944406] Lustre: Skipped 6 previous similar messages [ 8905.183349] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8905.543952] Lustre: server umount lustre-MDT0001 complete [ 8910.754770] Lustre: server umount lustre-OST0000 complete [ 8915.904324] Lustre: server umount lustre-OST0001 complete [ 8925.127961] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8938.317535] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8986.932966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8987.163768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8987.185801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8987.252518] Lustre: lustre-MDT0000: new disk, initializing [ 8987.321556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8987.327417] Lustre: Skipped 1 previous similar message [ 8987.345975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8993.898782] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9007.213695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9007.489222] Lustre: 212467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9007.614498] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9007.619217] Lustre: Skipped 1 previous similar message [ 9007.799523] Lustre: lustre-MDT0001: new disk, initializing [ 9008.010407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9008.040751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9013.249292] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9026.074953] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9034.075286] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9041.701303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9042.148504] Lustre: lustre-OST0000: new disk, initializing [ 9042.160622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9042.181552] Lustre: 214435:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9044.263781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9044.290996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9044.541512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9051.225374] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9062.583974] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9068.904923] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9069.249166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9074.216719] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9074.423825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9080.293651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9080.294329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9080.302256] LustreError: Skipped 1 previous similar message [ 9080.303218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9080.303223] Lustre: Skipped 15 previous similar messages [ 9080.329667] Lustre: Skipped 1 previous similar message [ 9082.607634] Lustre: server umount lustre-OST0000 complete [ 9090.018631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9095.655020] LustreError: 212480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9095.691437] LustreError: 212480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 22 previous similar messages [ 9095.917554] Lustre: server umount lustre-MDT0000 complete [ 9100.168556] LustreError: 214446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423980 with bad export cookie 2642256282671772288 [ 9100.186553] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9100.189546] LustreError: 214446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9100.477465] Lustre: server umount lustre-MDT0001 complete [ 9105.157283] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:53:03 (1786423983) [ 9126.184958] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9139.587986] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9183.050613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9183.488944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9183.567280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9183.666880] Lustre: lustre-MDT0000: new disk, initializing [ 9183.813435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9190.769696] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9204.863732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9205.065397] Lustre: 219049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9205.337186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9210.776515] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9222.000443] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9228.487383] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9236.602436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9236.982675] Lustre: 221024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9238.911970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9239.106430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9244.271381] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9256.018216] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9262.200419] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9262.531730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9269.481414] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9269.831957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9274.857445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9274.857663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9274.881986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9274.882027] Lustre: Skipped 3 previous similar messages [ 9274.884523] LustreError: Skipped 1 previous similar message [ 9274.960285] Lustre: Skipped 3 previous similar messages [ 9278.880908] Lustre: server umount lustre-OST0000 complete [ 9289.451458] Lustre: server umount lustre-MDT0000 complete [ 9290.213110] LustreError: 219062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9290.239293] LustreError: 219062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9293.726541] LustreError: 219043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424174 with bad export cookie 2642256282671773170 [ 9293.736171] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9293.737490] LustreError: 219043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9294.023558] Lustre: server umount lustre-MDT0001 complete [ 9304.407713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9304.831850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9304.837233] Lustre: Skipped 5 previous similar messages [ 9309.467660] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9319.416188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9323.686318] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9330.490634] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9336.843768] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9344.396469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9351.904970] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9360.512620] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9371.617650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9377.453126] Lustre: server umount lustre-OST0000 complete [ 9387.868189] Lustre: server umount lustre-MDT0000 complete [ 9391.855700] LustreError: 223225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424272 with bad export cookie 2642256282671774052 [ 9391.869575] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9391.907743] LustreError: 223225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9392.234425] Lustre: server umount lustre-MDT0001 complete [ 9501.816105] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 9504.991581] Key type lgssc unregistered [ 9505.430610] LNet: 226908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9505.446979] LNetError: 226908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9506.469739] LNet: Removed LNI 192.168.204.142@tcp [ 9507.541458] Key type .llcrypt unregistered [ 9507.545656] Key type ._llcrypt unregistered [ 9541.604283] Key type ._llcrypt registered [ 9541.611412] Key type .llcrypt registered [ 9541.793798] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9560.532348] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9562.115866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9562.176179] alg: No test for adler32 (adler32-zlib) [ 9563.315768] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [ 9563.515742] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 9565.151178] Key type lgssc registered [ 9566.554710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9608.026381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9608.089936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9609.460360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9609.519479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9609.584239] Lustre: lustre-MDT0000: new disk, initializing [ 9609.656368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9609.672268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9614.334663] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9626.888172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9627.028367] Lustre: 230278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9627.087932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9627.094618] Lustre: Skipped 1 previous similar message [ 9627.166684] Lustre: lustre-MDT0001: new disk, initializing [ 9627.273291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9627.306063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9627.318814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9633.350228] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9646.249671] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9654.331403] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9663.421289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9663.723475] Lustre: lustre-OST0000: new disk, initializing [ 9663.734502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9663.746791] Lustre: 232249:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9663.865332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9672.505709] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9673.377781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9673.404882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9673.737834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9684.182203] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9690.420939] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9690.894377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9696.957996] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9697.192029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9699.300418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9699.301705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9699.312627] LustreError: Skipped 1 previous similar message [ 9699.317195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9699.321259] Lustre: Skipped 1 previous similar message [ 9699.358112] Lustre: Skipped 1 previous similar message [ 9704.422651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9704.433591] Lustre: Skipped 1 previous similar message [ 9705.182821] Lustre: server umount lustre-OST0000 complete [ 9709.033696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9709.037859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9709.060023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9714.657357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9714.662949] Lustre: Skipped 2 previous similar messages [ 9719.779541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9719.790218] Lustre: Skipped 1 previous similar message [ 9723.359222] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9723.604188] Lustre: server umount lustre-MDT0000 complete [ 9727.897176] LustreError: 230270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424608 with bad export cookie 2180487934627222530 [ 9727.903294] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9727.919458] LustreError: 230270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9728.339751] Lustre: server umount lustre-MDT0001 complete [ 9750.001995] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9761.271450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9761.549160] LustreError: 235187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9761.572684] LustreError: 235187:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9761.615744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9766.082461] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9766.888669] LustreError: 235188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9772.015030] LustreError: 235187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9775.851113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9776.165202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9781.737693] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9785.402471] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9792.359523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9792.641298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9799.934623] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9809.463535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9809.702302] Lustre: lustre-OST0001: new disk, initializing [ 9809.705922] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9809.712231] Lustre: 237493:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9809.841476] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9815.960361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9815.999352] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9816.348355] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9819.246902] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9831.283423] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9837.645948] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9852.599914] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 01:05:30 (1786424730) [ 9857.001699] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9857.005814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9857.021041] Lustre: Skipped 3 previous similar messages [ 9857.032749] Lustre: Skipped 3 previous similar messages [ 9869.279109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9869.556324] Lustre: server umount lustre-MDT0000 complete [ 9872.355830] LustreError: 236355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9872.397287] LustreError: 236355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9874.654299] LustreError: 235167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424755 with bad export cookie 2180487934627223412 [ 9874.655749] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9874.673860] LustreError: 235167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9875.180051] Lustre: server umount lustre-MDT0001 complete [ 9888.826111] Lustre: server umount lustre-OST0000 complete [ 9892.832698] Lustre: 228490:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786424757/real 1786424757] req@ffff9f79e4cfce00 x1873201796863488/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786424773 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9892.923650] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9892.948115] Lustre: Skipped 1 previous similar message [ 9893.686651] Lustre: server umount lustre-OST0001 complete [ 9901.892405] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9913.653106] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9953.315801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9953.516432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9953.563782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9953.665652] Lustre: lustre-MDT0000: new disk, initializing [ 9953.773428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9953.789576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9959.046644] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9970.657330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9970.888273] Lustre: 241444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9970.993441] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9971.003615] Lustre: Skipped 1 previous similar message [ 9971.090203] Lustre: lustre-MDT0001: new disk, initializing [ 9971.308541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9971.328188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9977.862851] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9987.893372] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9994.704841] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10003.949361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10004.260309] Lustre: lustre-OST0000: new disk, initializing [10004.263216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10004.272231] Lustre: 243413:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10006.278599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10006.311380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10006.698442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10012.806429] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10023.214933] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10028.617056] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10029.033660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10035.229992] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10035.741263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10042.346554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10042.349940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10042.355450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10042.355459] Lustre: Skipped 8 previous similar messages [10042.404742] Lustre: Skipped 1 previous similar message [10044.593745] Lustre: server umount lustre-OST0000 complete [10052.583277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10055.428872] Lustre: server umount lustre-MDT0000 complete [10057.697178] LustreError: 241449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10057.713681] LustreError: 241449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10059.192807] LustreError: 241437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424939 with bad export cookie 2180487934627225547 [10059.205446] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10059.212920] LustreError: 241437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10059.586168] Lustre: server umount lustre-MDT0001 complete [10069.348897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10069.727282] LustreError: 245629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10069.811341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10069.818306] Lustre: Skipped 2 previous similar messages [10074.696590] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10083.458893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10089.212348] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10097.275843] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10104.404822] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10112.995572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10121.630147] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10130.932160] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10277.347749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10277.359784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10277.379066] Lustre: Skipped 1 previous similar message [10277.390274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10277.400296] Lustre: Skipped 3 previous similar messages [10283.299081] Lustre: server umount lustre-OST0000 complete [10288.610061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10288.619908] LustreError: Skipped 1 previous similar message [10293.244777] Lustre: server umount lustre-MDT0000 complete [10295.778094] LustreError: 245629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10295.811209] LustreError: 245629:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10297.936474] LustreError: 247473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425178 with bad export cookie 2180487934627226422 [10297.938602] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10297.947288] LustreError: 247473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10298.291714] Lustre: server umount lustre-MDT0001 complete [10401.939904] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [10404.918621] Key type lgssc unregistered [10405.403895] LNet: 249243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10405.413335] LNetError: 249243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10405.432822] LNet: Removed LNI 192.168.204.142@tcp [10406.605172] Key type .llcrypt unregistered [10406.616534] Key type ._llcrypt unregistered [10438.848329] Key type ._llcrypt registered [10438.850758] Key type .llcrypt registered [10439.033873] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [10459.985767] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10461.288443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10461.531079] alg: No test for adler32 (adler32-zlib) [10462.846317] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [10463.210221] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [10464.943264] Key type lgssc registered [10466.285663] Lustre: Echo OBD driver; http://www.lustre.org/ [10525.831402] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10537.245748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10537.314548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10538.702629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10538.748126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10538.880748] Lustre: lustre-MDT0000: new disk, initializing [10539.059512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10539.089212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10544.618962] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10555.193429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10555.413692] Lustre: 253319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10555.471403] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10555.472329] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10555.478886] Lustre: Skipped 1 previous similar message [10555.583403] Lustre: lustre-MDT0001: new disk, initializing [10555.763260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10555.819794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10555.834768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10560.540530] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10565.713736] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10572.778636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10573.032913] Lustre: lustre-OST0000: new disk, initializing [10573.036941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10573.043430] Lustre: 254948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10573.155567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10576.186489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10576.211486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10576.341951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10580.092679] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10591.066965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10591.240603] Lustre: lustre-OST0001: new disk, initializing [10591.248670] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10591.264629] Lustre: 255820:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10591.435529] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10598.094513] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10599.519509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10599.536309] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10599.650251] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10610.659611] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10618.302152] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10630.635792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10630.638313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10630.650341] Lustre: Skipped 3 previous similar messages [10630.654232] Lustre: Skipped 2 previous similar messages [10635.762586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10635.777272] Lustre: Skipped 4 previous similar messages [10636.532186] Lustre: server umount lustre-MDT0000 complete [10639.798989] LustreError: 256591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425520 with bad export cookie 16209811889568212567 [10639.806205] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10639.810834] LustreError: 256591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10640.169640] Lustre: server umount lustre-MDT0001 complete [10654.003713] Lustre: server umount lustre-OST0000 complete [10656.991116] Lustre: 250820:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786425521/real 1786425521] req@ffff9f78c804a300 x1873202740539008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786425537 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10657.047348] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10658.542378] Lustre: server umount lustre-OST0001 complete [10675.334380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10685.428138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10695.704200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10707.693635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10730.468436] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10742.891634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10742.911814] Lustre: lustre-MDT0000: reset Object Index mappings [10743.276616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10748.124586] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10748.391970] LustreError: 260834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10748.416121] LustreError: 260834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10753.511838] LustreError: 260833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10757.322146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10757.338318] Lustre: lustre-MDT0001: reset Object Index mappings [10757.695279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10762.804715] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10766.162607] Lustre: 261976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10775.141752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10775.470651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10782.643249] LustreError: 262330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10784.683383] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10787.749337] LustreError: 262331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10787.771151] LustreError: 262331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10787.792166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10792.946602] LustreError: 262331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10792.966944] LustreError: 262331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10795.298190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10800.648912] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10802.976894] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10811.071960] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10815.349824] Lustre: 263848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10821.089400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10821.127037] Lustre: Skipped 3 previous similar messages [10821.137590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10821.147654] Lustre: Skipped 3 previous similar messages [10822.636828] Lustre: server umount lustre-MDT0000 complete [10826.219386] LustreError: 260834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10826.255389] LustreError: 260834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10826.545275] LustreError: 260813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425706 with bad export cookie 16209811889568219364 [10826.555585] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10826.585045] LustreError: 260813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10827.016132] Lustre: server umount lustre-MDT0001 complete [10841.718867] Lustre: server umount lustre-OST0000 complete [10855.831023] Lustre: server umount lustre-OST0001 complete [10878.977440] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 01:22:38 (1786425758) [10899.930466] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [10914.224794] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10955.111713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10955.601842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10955.657402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10955.785618] Lustre: lustre-MDT0000: new disk, initializing [10955.996813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10956.012274] Lustre: Skipped 1 previous similar message [10956.053988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10961.316414] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10972.085227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10972.259348] Lustre: 268106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10972.328991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10972.334800] Lustre: Skipped 1 previous similar message [10972.420434] Lustre: lustre-MDT0001: new disk, initializing [10972.553300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10972.568848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10977.799309] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10990.337826] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10996.861324] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11004.419749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11004.714849] Lustre: lustre-OST0000: new disk, initializing [11004.717748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11004.723659] Lustre: 270075:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11006.600050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11006.628342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11006.779302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11011.358659] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11022.942514] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11029.038848] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11029.283652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11034.191417] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11034.408464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11037.669419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11037.676177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11037.691080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11037.701833] LustreError: Skipped 1 previous similar message [11037.712089] Lustre: Skipped 1 previous similar message [11042.809266] LustreError: 270058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11042.933809] Lustre: server umount lustre-OST0000 complete [11047.904899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11047.918469] Lustre: Skipped 2 previous similar messages [11047.919665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11047.929779] Lustre: Skipped 1 previous similar message [11053.841401] Lustre: server umount lustre-MDT0000 complete [11058.086207] LustreError: 268097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425938 with bad export cookie 16209811889568221450 [11058.102721] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11058.104209] LustreError: 268097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11058.395467] Lustre: server umount lustre-MDT0001 complete [11082.800238] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11094.589316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11094.856440] LustreError: 273010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11094.870853] LustreError: 273010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11094.900927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11094.906802] Lustre: Skipped 2 previous similar messages [11100.423532] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11110.458866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11116.461849] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11120.021450] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11128.342366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11135.566565] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11145.551249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11145.799374] Lustre: lustre-OST0001: new disk, initializing [11145.803724] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11145.810370] Lustre: 275318:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11147.038291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11147.070058] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11147.416693] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11154.232786] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11164.682202] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11168.819766] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11175.893179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11175.920287] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11182.561355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11182.572022] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11182.587931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11182.596035] Lustre: Skipped 2 previous similar messages [11196.383156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11196.770968] Lustre: server umount lustre-MDT0000 complete [11198.448309] LustreError: 273010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11198.476606] LustreError: 273010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [11201.142946] LustreError: 272992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786426081 with bad export cookie 16209811889568222339 [11201.169951] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11201.173270] LustreError: 272992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11201.792670] Lustre: server umount lustre-MDT0001 complete [11215.704582] Lustre: server umount lustre-OST0000 complete [11230.086723] Lustre: server umount lustre-OST0001 complete [11250.255620] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11263.431037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11263.762242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11263.771309] Lustre: Skipped 3 previous similar messages [11270.536623] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11281.015904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11288.281383] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11291.645439] Lustre: 279516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11300.793853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11307.962547] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11310.503843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11318.185588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11323.932822] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11327.229751] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11336.614855] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11340.871667] Lustre: 281394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11347.354468] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11347.427684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11360.338870] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 01:30:38 (1786426238) [11378.656166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11378.670943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11378.685155] Lustre: Skipped 3 previous similar messages [11378.689849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11378.702030] Lustre: Skipped 12 previous similar messages [11383.358550] Lustre: server umount lustre-MDT0000 complete [11385.322525] LustreError: 280405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11385.356653] LustreError: 280405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [11388.185745] LustreError: 279118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786426268 with bad export cookie 16209811889568228632 [11388.202458] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11388.226974] LustreError: 279118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11394.735996] Lustre: server umount lustre-MDT0001 complete [11405.237956] Lustre: server umount lustre-OST0000 complete [11410.498402] Lustre: server umount lustre-OST0001 complete [11420.221522] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [11435.480605] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11493.795988] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11506.878477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11507.140322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11507.162696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11507.226882] Lustre: lustre-MDT0000: new disk, initializing [11507.307644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11511.668179] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11524.265396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11524.518663] Lustre: 285659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11524.934847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11524.942478] Lustre: Skipped 4 previous similar messages [11524.990140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11530.809886] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11544.698427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11545.091740] Lustre: 287292:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11547.475725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11553.550651] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11565.138916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11565.394203] Lustre: 288163:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11567.091973] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11567.112933] Lustre: Skipped 1 previous similar message [11567.386383] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11574.435470] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11585.397681] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11589.789179] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11589.799185] Lustre: Skipped 1 previous similar message [11608.221360] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 01:34:46 (1786426486) [11612.129474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11612.141344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11612.156388] Lustre: Skipped 5 previous similar messages [11612.165682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11612.170451] Lustre: Skipped 8 previous similar messages [11617.246486] Lustre: server umount lustre-MDT0000 complete [11621.732744] LustreError: 285650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786426502 with bad export cookie 16209811889568236514 [11621.743730] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11621.748916] LustreError: 285650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11622.065839] Lustre: server umount lustre-MDT0001 complete [11637.121257] Lustre: server umount lustre-OST0000 complete [11640.287710] Lustre: 250820:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786426504/real 1786426504] req@ffff9f79e5225880 x1873202740860160/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786426520 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11641.236722] Lustre: server umount lustre-OST0001 complete [11651.362532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11651.891483] LustreError: 290409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11651.926671] LustreError: 290409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [11656.894815] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11667.375777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11672.406945] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11681.537454] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11688.259628] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11697.293167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11706.623418] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11717.090606] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11735.933115] loop1: detected capacity change from 0 to 204800000 [11736.295501] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11736.320412] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11736.342020] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11741.943498] loop1: detected capacity change from 0 to 204800000 [11748.847444] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11748.933022] LustreError: 290399:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs17037: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11748.955341] LustreError: 290399:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs17037 [11748.960420] LustreError: 290399:0:(mgs_handler.c:517:mgs_target_reg()) fs17037-MDT0001: Error checking target index: rc = -13 [11748.975763] LustreError: fs17037-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11748.996892] LustreError: 294133:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11749.002801] LustreError: 294133:0:(tgt_mount.c:2129:server_put_super()) no obd fs17037-MDT0001 [11749.006958] LustreError: 294133:0:(tgt_mount.c:116:server_deregister_mount()) fs17037-MDT0001 not registered [11749.045857] Lustre: server umount fs17037-MDT0001 complete [11749.048510] LustreError: 294133:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11759.252734] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11759.325665] Lustre: Setting parameter fs17037-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs17037-MDT0001 [11759.359617] Lustre: srv-fs17037-MDT0001: No data found on store. Initialize space. [11759.363249] Lustre: Skipped 4 previous similar messages [11759.420157] Lustre: fs17037-MDT0001: new disk, initializing [11759.422847] Lustre: Skipped 3 previous similar messages [11759.520147] Lustre: srv-fs17037-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11759.531800] Lustre: 294599:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17037-MDT0001-mdtlov: can't create new llog: rc = -115 [11759.546589] LustreError: 294599:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17037-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11765.738744] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11774.183211] Lustre: server umount fs17037-MDT0001 complete [11780.069075] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11789.702842] loop1: detected capacity change from 0 to 204800000 [11795.478576] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11797.231944] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11797.493147] Lustre: 296170:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17037-MDT0001-mdtlov: can't create new llog: rc = -115 [11797.505212] LustreError: 296170:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17037-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11803.258635] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11808.627962] Lustre: server umount fs17037-MDT0001 complete [11820.570473] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [11820.586516] Lustre: Skipped 4 previous similar messages [11821.026809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11821.055242] Lustre: Skipped 6 previous similar messages [11825.733575] Lustre: server umount lustre-MDT0000 complete [11838.257212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11838.396752] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11844.121279] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11846.846434] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11857.026397] loop1: detected capacity change from 0 to 204800000 [11865.299584] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11865.966074] Lustre: 298619:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17037-MDT0001-mdtlov: can't create new llog: rc = -115 [11865.983080] LustreError: 298619:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17037-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11871.604611] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11877.895664] Lustre: server umount fs17037-MDT0001 complete [11905.507646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11911.827935] Lustre: server umount lustre-OST0000 complete [11922.794501] Lustre: server umount lustre-MDT0000 complete [11927.011763] LustreError: 290390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786426807 with bad export cookie 16209811889568239195 [11927.016244] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11927.044982] LustreError: 290390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11927.612170] Lustre: server umount lustre-MDT0001 complete [11944.964622] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [11950.113925] Key type lgssc unregistered [11950.745919] LNet: 300969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11950.753678] LNetError: 300969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11950.773454] LNet: Removed LNI 192.168.204.142@tcp [11952.397184] Key type .llcrypt unregistered [11952.408228] Key type ._llcrypt unregistered [11982.763776] Key type ._llcrypt registered [11982.767474] Key type .llcrypt registered [11983.091452] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11984.541672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11984.707555] alg: No test for adler32 (adler32-zlib) [11986.253072] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [11986.735057] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [11988.519241] Key type lgssc registered [11990.230613] Lustre: Echo OBD driver; http://www.lustre.org/ [12000.736791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12000.758561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12002.294647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12007.419119] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12012.520904] LustreError: 302243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12012.568249] LustreError: 302243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12017.508179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12017.640159] LustreError: 302244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12018.003939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12024.395171] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12033.156508] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12040.602385] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12049.999651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12050.471144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12058.776143] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12062.767160] LustreError: 304575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.830912] LustreError: 304073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12068.316787] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12071.977798] LustreError: 304575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12071.995431] LustreError: 304575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12080.118114] LustreError: 304073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12080.182720] LustreError: 304073:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12094.080912] loop1: detected capacity change from 0 to 204800000 [12094.376651] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12094.386257] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12094.401241] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12097.509549] LustreError: 304075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12097.524662] LustreError: 304075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [12100.924694] loop1: detected capacity change from 0 to 204800000 [12107.374672] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12107.548417] Lustre: fs20681-OST0000: new disk, initializing [12107.552327] Lustre: srv-fs20681-OST0000: No data found on store. Initialize space. [12107.558780] Lustre: 305982:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12107.640679] Lustre: fs20681-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12114.822167] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12123.573729] Lustre: server umount fs20681-OST0000 complete [12131.333894] LustreError: 304074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12131.366513] LustreError: 304074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [12137.765669] loop1: detected capacity change from 0 to 204800000 [12143.594437] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12144.225251] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12144.466615] Lustre: fs20681-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12148.709358] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12150.986819] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12161.346416] loop2: detected capacity change from 0 to 204800000 [12161.699029] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12161.730465] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12161.749976] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12168.058988] loop2: detected capacity change from 0 to 204800000 [12175.138894] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12175.203304] LustreError: fs20681-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12175.215313] LustreError: fs20681-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12175.224394] LustreError: 309025:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12175.232979] LustreError: 309025:0:(tgt_mount.c:2129:server_put_super()) no obd fs20681-OST0001 [12175.241397] LustreError: 309025:0:(tgt_mount.c:116:server_deregister_mount()) fs20681-OST0001 not registered [12175.284814] Lustre: server umount fs20681-OST0001 complete [12175.287777] LustreError: 309025:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [12186.542756] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12186.638898] Lustre: fs20681-OST0001: new disk, initializing [12186.643332] Lustre: srv-fs20681-OST0001: No data found on store. Initialize space. [12186.649109] Lustre: 309576:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12186.720368] Lustre: fs20681-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12195.063450] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12197.871539] LustreError: 304075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12197.897120] LustreError: 304075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [12205.248391] Lustre: server umount fs20681-OST0000 complete [12216.260188] Lustre: server umount fs20681-OST0001 complete [12231.196831] loop1: detected capacity change from 0 to 204800000 [12237.206765] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12237.458526] Lustre: fs20681-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12244.720894] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12251.805331] loop2: detected capacity change from 0 to 204800000 [12257.668524] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12266.304234] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12277.978669] Lustre: server umount fs20681-OST0000 complete [12288.926070] Lustre: server umount fs20681-OST0001 complete [12295.160497] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12300.279307] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12305.378704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12305.394686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12305.404102] Lustre: Skipped 1 previous similar message [12310.365414] Lustre: server umount lustre-OST0000 complete [12315.107192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12315.116955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12315.130909] Lustre: Skipped 1 previous similar message [12315.138306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12320.741059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12320.758446] Lustre: Skipped 2 previous similar messages [12321.061791] Lustre: server umount lustre-MDT0000 complete [12325.167484] LustreError: 306727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786427205 with bad export cookie 1736566418376248962 [12325.187907] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12325.201951] LustreError: 306727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12325.753977] Lustre: server umount lustre-MDT0001 complete [12341.860801] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12345.573564] Key type lgssc unregistered [12346.008790] LNet: 315505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12346.017577] LNetError: 315505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12347.046706] LNet: Removed LNI 192.168.204.142@tcp [12348.438506] Key type .llcrypt unregistered [12348.441202] Key type ._llcrypt unregistered [12387.416719] Key type ._llcrypt registered [12387.422310] Key type .llcrypt registered [12387.558463] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [12407.473533] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12409.309427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12409.325982] alg: No test for adler32 (adler32-zlib) [12410.801659] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [12411.044677] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12412.735397] Key type lgssc registered [12414.573756] Lustre: Echo OBD driver; http://www.lustre.org/ [12459.289804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12459.338883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12460.601470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12460.649431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12460.737956] Lustre: lustre-MDT0000: new disk, initializing [12460.829658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12460.862717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12466.977280] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12481.412517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12481.784782] Lustre: 319370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12481.928127] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12481.936405] Lustre: Skipped 1 previous similar message [12482.009983] Lustre: lustre-MDT0001: new disk, initializing [12482.108188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12482.162257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12482.189954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12488.062404] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12500.021524] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12507.430842] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12517.714312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12518.150204] Lustre: lustre-OST0000: new disk, initializing [12518.152973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12518.164502] Lustre: 321338:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12518.257049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12525.941384] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12526.266030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12526.275235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12526.513412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12537.174932] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12540.242976] Lustre: Mounted lustre-client - version 2.17.56_50_gb0a9653 [12541.452679] Lustre: Unmounted lustre-client [12555.751693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12555.769683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12555.782506] Lustre: Skipped 1 previous similar message [12555.795376] Lustre: Skipped 1 previous similar message [12560.316916] Lustre: server umount lustre-OST0000 complete [12565.989458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12565.993423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12566.004595] Lustre: Skipped 1 previous similar message [12566.015000] Lustre: Skipped 1 previous similar message [12570.805370] Lustre: server umount lustre-MDT0000 complete [12575.202772] LustreError: 319361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786427455 with bad export cookie 12821342272485480537 [12575.206814] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12575.213303] LustreError: 319361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12575.461615] Lustre: server umount lustre-MDT0001 complete [12591.996559] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12593.378211] LNet: 1 local NIs in recovery (showing 1): 192.168.204.142@tcp [12595.992880] Key type lgssc unregistered [12596.384304] LNet: 323505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12596.397181] LNetError: 323505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12670.181914] LNet: Removed LNI 192.168.204.142@tcp [12671.524407] Key type .llcrypt unregistered [12671.527924] Key type ._llcrypt unregistered [12694.269710] Key type ._llcrypt registered [12694.273180] Key type .llcrypt registered [12694.388530] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [12713.502628] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12714.435650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12714.511161] alg: No test for adler32 (adler32-zlib) [12715.758001] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [12716.038707] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12717.703459] Key type lgssc registered [12719.536225] Lustre: Echo OBD driver; http://www.lustre.org/ [12756.781442] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:53:55 (1786427635) [12764.848296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12764.883779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12766.121181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12766.145510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12766.214930] Lustre: lustre-MDT0000: new disk, initializing [12766.282969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12766.300883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12772.788470] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12784.872488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12785.002151] Lustre: 326732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12785.062721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12785.067446] Lustre: Skipped 1 previous similar message [12785.134942] Lustre: lustre-MDT0001: new disk, initializing [12785.258921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12785.320917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12785.344578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12791.082734] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12802.676462] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12808.967678] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12817.167593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12817.453735] Lustre: lustre-OST0000: new disk, initializing [12817.458939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12817.467673] Lustre: 328698:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12817.591329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12824.142093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12824.151536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12824.271945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12824.404249] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12834.958618] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12855.273416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12855.273765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12855.277554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12855.285197] LustreError: Skipped 1 previous similar message [12855.317686] Lustre: Skipped 1 previous similar message [12858.670776] Lustre: server umount lustre-OST0000 complete [12865.506747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12865.510777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12865.532146] Lustre: Skipped 1 previous similar message [12865.553676] Lustre: Skipped 2 previous similar messages [12869.436142] Lustre: server umount lustre-MDT0000 complete [12874.276892] LustreError: 326724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786427754 with bad export cookie 2042889530633781238 [12874.293425] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12874.295475] LustreError: 326724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12874.689433] Lustre: server umount lustre-MDT0001 complete [12889.131601] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12893.456211] Key type lgssc unregistered [12893.896822] LNet: 330612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12893.908193] LNetError: 330612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12893.933305] LNet: Removed LNI 192.168.204.142@tcp [12894.870367] Key type .llcrypt unregistered [12894.872798] Key type ._llcrypt unregistered [12926.162968] Key type ._llcrypt registered [12926.165546] Key type .llcrypt registered [12926.296151] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12928.217240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12928.241064] alg: No test for adler32 (adler32-zlib) [12929.899863] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [12930.378759] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12932.167608] Key type lgssc registered [12934.194901] Lustre: Echo OBD driver; http://www.lustre.org/ [12944.564556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12944.586549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12946.302868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12952.382952] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12955.616924] LustreError: 331894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12955.631563] LustreError: 331894:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12960.749769] LustreError: 331895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12963.436121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12963.889547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12969.473882] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12977.947627] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12985.204050] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12995.011944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12995.417968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13002.351109] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13011.247310] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13028.775223] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13031.006106] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13036.514405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13036.530744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13036.556070] Lustre: Skipped 2 previous similar messages [13036.570771] Lustre: Skipped 2 previous similar messages [13038.618183] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [13041.640767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13041.650362] Lustre: Skipped 2 previous similar messages [13043.698584] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [13048.819207] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [13048.834731] Lustre: Skipped 3 previous similar messages [13050.335178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13050.500755] Lustre: server umount lustre-MDT0000 complete [13051.878754] LustreError: 333739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13051.900323] LustreError: 333739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13053.938381] LustreError: 331889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13054.546895] LustreError: 331876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786427934 with bad export cookie 4566245053508882501 [13054.555230] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13054.568056] LustreError: 331876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13056.994766] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13057.014656] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13059.063631] LustreError: 333193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13059.086897] LustreError: 333193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13060.969954] Lustre: server umount lustre-MDT0001 complete [13065.250854] Lustre: server umount lustre-OST0000 complete [13075.369766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13081.339539] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13084.913571] Lustre: 336102:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.142@tcp [13087.773725] Lustre: server umount MGS complete [13098.049837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13098.535587] LustreError: 336674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13098.559486] LustreError: 336674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13098.664672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13103.997839] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13113.282820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13113.536743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13118.693946] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13127.455863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13127.739348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13133.902508] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13140.656449] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13142.539750] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13148.643922] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13150.402964] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13155.802686] Lustre: server umount lustre-OST0000 complete [13159.906352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13159.915718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13159.940182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13159.947591] Lustre: Skipped 2 previous similar messages [13164.523493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13170.802773] Lustre: server umount lustre-MDT0000 complete [13173.728432] Lustre: 331205:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786428038/real 1786428038] req@000000007c526448 x1873205327364480/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786428054 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13173.774827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13173.797905] LustreError: 339022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13173.829190] LustreError: 339022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [13174.751170] Lustre: 331207:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786428039/real 1786428039] req@00000000692fed87 x1873205327365248/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786428055 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13175.133985] LustreError: 337408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786428055 with bad export cookie 4566245053508883341 [13175.146447] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13175.148153] LustreError: 337408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13175.452484] Lustre: server umount lustre-MDT0001 complete [13187.441472] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13190.630474] Key type lgssc unregistered [13190.863957] LNet: 340128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13190.869318] LNetError: 340128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13190.883680] LNet: Removed LNI 192.168.204.142@tcp [13191.592689] Key type .llcrypt unregistered [13191.596304] Key type ._llcrypt unregistered [13210.652323] Key type ._llcrypt registered [13210.657327] Key type .llcrypt registered [13210.815669] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [13226.902891] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [13227.943568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13227.963720] alg: No test for adler32 (adler32-zlib) [13229.758704] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [13230.002817] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [13231.679232] Key type lgssc registered [13232.721408] Lustre: Echo OBD driver; http://www.lustre.org/ [13268.307792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13268.329372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13269.514178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13269.541378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13269.608637] Lustre: lustre-MDT0000: new disk, initializing [13269.679072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13269.710948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13273.608436] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13284.712706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13284.829566] Lustre: 343289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13284.881324] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13284.884795] Lustre: Skipped 1 previous similar message [13284.963544] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13285.031339] Lustre: lustre-MDT0001: new disk, initializing [13285.214354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13285.263951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13285.286449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13290.054707] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13300.572202] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13306.721903] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13313.066102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13313.286537] Lustre: lustre-OST0000: new disk, initializing [13313.290119] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13313.297601] Lustre: 345258:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13313.389484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13319.865988] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13320.344928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13320.352640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13320.542481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13329.974251] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13334.611900] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13334.904552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13339.886374] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13340.066967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13346.276433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13346.276613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13346.289773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13346.292526] Lustre: Skipped 1 previous similar message [13346.302099] Lustre: Skipped 1 previous similar message [13348.300077] Lustre: server umount lustre-OST0000 complete [13351.904736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13351.909174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13351.919273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13356.518694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13356.524057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13361.633124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13361.643447] Lustre: Skipped 2 previous similar messages [13366.239208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13366.430024] Lustre: server umount lustre-MDT0000 complete [13370.046214] LustreError: 343282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786428250 with bad export cookie 1495999159245620332 [13370.051640] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13370.066867] LustreError: 343282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13370.538462] Lustre: server umount lustre-MDT0001 complete [13381.624309] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:04:20 (1786428260) [13401.078987] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [13413.018816] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [13454.890018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13455.152674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13455.204130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13455.276510] Lustre: lustre-MDT0000: new disk, initializing [13455.365468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13455.380784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13459.947496] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13471.158704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13471.271903] Lustre: 350023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13471.318823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13471.322861] Lustre: Skipped 1 previous similar message [13471.377889] Lustre: lustre-MDT0001: new disk, initializing [13471.453101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13471.502392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13471.509792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13476.603029] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13486.319594] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13493.050397] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13500.521482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13500.752491] Lustre: lustre-OST0000: new disk, initializing [13500.756693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13500.762702] Lustre: 351993:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13500.857878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13509.735234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13509.765988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13509.873346] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13510.059943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13520.402788] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13561.315493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13561.318192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13561.333425] LustreError: Skipped 1 previous similar message [13561.337458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13561.343792] Lustre: Skipped 1 previous similar message [13561.355232] Lustre: Skipped 3 previous similar messages [13562.585224] Lustre: server umount lustre-OST0000 complete [13568.993772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13569.005084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13573.247928] Lustre: server umount lustre-MDT0000 complete [13578.257796] LustreError: 350016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786428458 with bad export cookie 1495999159245621214 [13578.260864] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13578.268810] LustreError: 350016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13578.551548] Lustre: server umount lustre-MDT0001 complete [13591.651908] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13594.582602] Key type lgssc unregistered [13594.893290] LNet: 354170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13594.907349] LNetError: 354170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13594.920232] LNet: Removed LNI 192.168.204.142@tcp [13595.805952] Key type .llcrypt unregistered [13595.808672] Key type ._llcrypt unregistered [13706.266959] Key type ._llcrypt registered [13706.273200] Key type .llcrypt registered [13706.417949] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13706.846387] Key type .llcrypt unregistered [13706.852778] Key type ._llcrypt unregistered [13797.861996] Key type ._llcrypt registered [13797.864578] Key type .llcrypt registered [13798.091412] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13798.642320] Key type .llcrypt unregistered [13798.646477] Key type ._llcrypt unregistered [13833.439761] Key type ._llcrypt registered [13833.442242] Key type .llcrypt registered [13833.521408] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13833.896562] Key type .llcrypt unregistered [13833.900993] Key type ._llcrypt unregistered [13869.902831] Key type ._llcrypt registered [13869.905678] Key type .llcrypt registered [13870.019606] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13870.307443] Key type .llcrypt unregistered [13870.311965] Key type ._llcrypt unregistered [13956.162844] Key type ._llcrypt registered [13956.165936] Key type .llcrypt registered [13956.271346] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13956.594553] Key type .llcrypt unregistered [13956.597900] Key type ._llcrypt unregistered [13995.596496] Key type ._llcrypt registered [13995.599739] Key type .llcrypt registered [13995.725782] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13996.157611] Key type .llcrypt unregistered [13996.164829] Key type ._llcrypt unregistered [14035.839543] Key type ._llcrypt registered [14035.846951] Key type .llcrypt registered [14036.108601] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [14036.717248] Key type .llcrypt unregistered [14036.723576] Key type ._llcrypt unregistered [14074.019904] Key type ._llcrypt registered [14074.022653] Key type .llcrypt registered [14074.124816] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [14074.441211] Key type .llcrypt unregistered [14074.445545] Key type ._llcrypt unregistered [14115.676563] Key type ._llcrypt registered [14115.682375] Key type .llcrypt registered [14116.000344] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [14116.512524] Key type .llcrypt unregistered [14116.515658] Key type ._llcrypt unregistered [14154.371906] Key type ._llcrypt registered [14154.374207] Key type .llcrypt registered [14154.581354] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [14155.124473] Key type .llcrypt unregistered [14155.130301] Key type ._llcrypt unregistered [14232.426486] Key type ._llcrypt registered [14232.439542] Key type .llcrypt registered [14232.690630] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [14233.362481] Key type .llcrypt unregistered [14233.369796] Key type ._llcrypt unregistered [14273.944803] Key type ._llcrypt registered [14273.947320] Key type .llcrypt registered [14274.061774] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [14274.354886] Key type .llcrypt unregistered [14274.357382] Key type ._llcrypt unregistered [14315.475617] Key type ._llcrypt registered [14315.478167] Key type .llcrypt registered [14315.573723] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [14315.944785] Key type .llcrypt unregistered [14315.950935] Key type ._llcrypt unregistered [14390.064623] Key type ._llcrypt registered [14390.070493] Key type .llcrypt registered [14390.310367] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [14411.903813] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [14414.989109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14415.037066] alg: No test for adler32 (adler32-zlib) [14417.042709] Lustre: Lustre: Build Version: 2.17.56_50_gb0a9653 [14417.716595] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [14419.615234] Key type lgssc registered [14421.929741] Lustre: Echo OBD driver; http://www.lustre.org/ [14470.312471] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14470.340819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14471.586379] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14471.635434] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14471.733578] Lustre: tst250fs-MDT0000: new disk, initializing [14471.875854] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14471.898230] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14478.270421] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [14490.547379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14490.744704] Lustre: 374587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14490.797609] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14490.801138] Lustre: Skipped 1 previous similar message [14490.851401] Lustre: tst250fs-MDT0001: new disk, initializing [14490.986619] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14491.019205] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14491.034448] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14496.677318] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [14502.715736] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14510.679083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14511.093205] Lustre: tst250fs-OST0000: new disk, initializing [14511.101223] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14511.114373] Lustre: 376220:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14511.246050] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14514.260368] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14514.288625] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14514.580673] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14521.401831] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [14533.021167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14533.190840] Lustre: tst250fs-OST0001: new disk, initializing [14533.194943] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14533.204733] Lustre: 377089:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14533.344679] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14541.084082] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14541.135199] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14541.363433] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14542.706620] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [14550.252370] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14552.459924] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14552.464741] Lustre: Skipped 1 previous similar message [14577.127751] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14577.135931] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14577.159239] Lustre: Skipped 3 previous similar messages [14577.171095] Lustre: Skipped 2 previous similar messages [14581.659049] Lustre: server umount tst250fs-MDT0000 complete [14585.973300] LustreError: 374578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786429466 with bad export cookie 3041989056363292128 [14585.974821] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14585.984351] LustreError: 374578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14586.502922] Lustre: server umount tst250fs-MDT0001 complete [14601.131403] Lustre: server umount tst250fs-OST0000 complete [14603.552568] Lustre: 372807:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786429467/real 1786429467] req@ffff9f79d7609f80 x1873206887652480/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786429483 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14603.594665] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14606.287233] Lustre: server umount tst250fs-OST0001 complete [14629.948304] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [14642.903943] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [14688.158483] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 02:26:06 (1786429566) [14689.965470] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14710.969199] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [14726.102538] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [14765.831035] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14388 sec ========= 02:27:24 (1786429644) [14769.519758] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:27:26 (1786429646) === [14773.930618] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:27:32 (1786429652) === [14826.214720] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [14829.437760] Key type lgssc unregistered [14829.898330] LNet: 386252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14829.908175] LNetError: 386252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14829.929913] LNet: Removed LNI 192.168.204.142@tcp [14831.209193] Key type .llcrypt unregistered [14831.213653] Key type ._llcrypt unregistered