[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474373135 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002502] x2apic enabled [ 0.003014] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007938] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008044] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009026] pid_max: default: 32768 minimum: 301 [ 0.010206] LSM: Security Framework initializing [ 0.011092] Yama: becoming mindful. [ 0.012058] SELinux: Initializing. [ 0.014041] *** VALIDATE selinux *** [ 0.024587] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029109] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032120] *** VALIDATE tmpfs *** [ 0.034245] *** VALIDATE proc *** [ 0.035283] *** VALIDATE cgroup *** [ 0.036013] *** VALIDATE cgroup2 *** [ 0.038237] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040002] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042032] Spectre V2 : User space: Vulnerable [ 0.043013] Speculative Store Bypass: Vulnerable [ 0.045776] debug: unmapping init [mem 0xffffffff96e59000-0xffffffff96e60fff] [ 0.047891] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048764] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049024] ... version: 2 [ 0.050017] ... bit width: 48 [ 0.051012] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053018] ... max period: 00007fffffffffff [ 0.054027] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.056367] rcu: Hierarchical SRCU implementation. [ 0.058474] smp: Bringing up secondary CPUs ... [ 0.059541] x86: Booting SMP configuration: [ 0.060021] .... node #0, CPUs: #1 #2 #3 [ 0.064102] smp: Brought up 1 node, 4 CPUs [ 0.066019] smpboot: Max logical packages: 1 [ 0.067019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141131] node 0 deferred pages initialised in 69ms [ 0.145139] devtmpfs: initialized [ 0.146225] x86/mm: Memory block size: 128MB [ 0.150747] gcov: version magic: 0x41383552 [ 0.154414] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159367] pinctrl core: initialized pinctrl subsystem [ 0.161161] [ 0.161673] ************************************************************* [ 0.163014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167013] ** ** [ 0.169015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172016] ** ** [ 0.174009] ** This means that this kernel is built to expose internal ** [ 0.175009] ** IOMMU data structures, which may compromise security on ** [ 0.177022] ** your system. ** [ 0.180014] ** ** [ 0.183014] ** If you see this message and you are not debugging the ** [ 0.185012] ** kernel, report this immediately to your vendor! ** [ 0.188012] ** ** [ 0.191012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193010] ************************************************************* [ 0.195714] NET: Registered protocol family 16 [ 0.198452] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.205458] cpuidle: using governor menu [ 0.207844] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.210424] PCI: Using configuration type 1 for base access [ 0.213127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.222150] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.224100] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227355] cryptd: max_cpu_qlen set to 1000 [ 0.229282] ACPI: Added _OSI(Module Device) [ 0.231014] ACPI: Added _OSI(Processor Device) [ 0.233010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239460] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246417] ACPI: Interpreter enabled [ 0.247064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.249013] ACPI: Using IOAPIC for interrupt routing [ 0.251116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255724] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265571] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.268047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.275090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.285411] acpiphp: Slot [2] registered [ 0.288196] acpiphp: Slot [5] registered [ 0.292174] acpiphp: Slot [6] registered [ 0.294176] acpiphp: Slot [7] registered [ 0.298198] acpiphp: Slot [8] registered [ 0.299137] acpiphp: Slot [9] registered [ 0.302225] acpiphp: Slot [10] registered [ 0.304133] acpiphp: Slot [3] registered [ 0.308239] acpiphp: Slot [4] registered [ 0.311148] acpiphp: Slot [11] registered [ 0.312000] acpiphp: Slot [12] registered [ 0.312000] acpiphp: Slot [13] registered [ 0.313136] acpiphp: Slot [14] registered [ 0.314080] acpiphp: Slot [15] registered [ 0.315100] acpiphp: Slot [16] registered [ 0.317177] acpiphp: Slot [17] registered [ 0.319119] acpiphp: Slot [18] registered [ 0.320194] acpiphp: Slot [19] registered [ 0.322127] acpiphp: Slot [20] registered [ 0.323114] acpiphp: Slot [21] registered [ 0.328145] acpiphp: Slot [22] registered [ 0.333234] acpiphp: Slot [23] registered [ 0.335206] acpiphp: Slot [24] registered [ 0.338229] acpiphp: Slot [25] registered [ 0.341214] acpiphp: Slot [26] registered [ 0.343138] acpiphp: Slot [27] registered [ 0.345124] acpiphp: Slot [28] registered [ 0.347147] acpiphp: Slot [29] registered [ 0.350217] acpiphp: Slot [30] registered [ 0.353120] acpiphp: Slot [31] registered [ 0.357096] PCI host bridge to bus 0000:00 [ 0.360023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.364026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.368029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.373032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.377262] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.380036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.387488] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.396071] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.404196] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.422000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.426000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.428018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.432022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.434014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.435000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.438927] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.446054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.450096] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 10742 usecs [ 0.457197] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.467024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.488016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.499191] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.509570] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.518019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.532020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.564017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.578000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.588016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.596000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.617017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.645920] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.661016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.667017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.697020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.718730] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.730031] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.739019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.767017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.775000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.784026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.795018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.823017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.852154] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.878018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.894015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.930045] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.959242] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.962444] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.964429] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.967383] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.969379] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.972000] iommu: Default domain type: Passthrough [ 0.973092] SCSI subsystem initialized [ 0.974000] ACPI: bus type USB registered [ 0.976175] usbcore: registered new interface driver usbfs [ 0.978185] usbcore: registered new interface driver hub [ 0.980117] usbcore: registered new device driver usb [ 0.982376] pps_core: LinuxPPS API ver. 1 registered [ 0.984010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.989060] PTP clock support registered [ 0.993155] EDAC MC: Ver: 3.0.0 [ 0.996131] PCI: Using ACPI for IRQ routing [ 0.997910] NetLabel: Initializing [ 1.000011] NetLabel: domain hash size = 128 [ 1.001010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.004082] NetLabel: unlabeled traffic allowed by default [ 1.007042] vgaarb: loaded [ 1.008818] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.011012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.019268] clocksource: Switched to clocksource kvm-clock [ 1.265083] VFS: Disk quotas dquot_6.6.0 [ 1.268265] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.272250] *** VALIDATE ramfs *** [ 1.275504] *** VALIDATE hugetlbfs *** [ 1.277833] pnp: PnP ACPI init [ 1.289094] pnp: PnP ACPI: found 6 devices [ 1.349397] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.360327] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.371720] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.377851] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.390267] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.395190] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.408791] NET: Registered protocol family 2 [ 1.422308] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.435164] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.459447] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.468967] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.474311] TCP: Hash tables configured (established 65536 bind 65536) [ 1.479829] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.482685] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.484754] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.486837] NET: Registered protocol family 1 [ 1.489415] RPC: Registered named UNIX socket transport module. [ 1.492727] RPC: Registered udp transport module. [ 1.494441] RPC: Registered tcp transport module. [ 1.495978] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.498413] NET: Registered protocol family 44 [ 1.501138] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.505211] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.508321] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.511544] PCI: CLS 0 bytes, default 64 [ 1.514613] Unpacking initramfs... [ 4.357119] debug: unmapping init [mem 0xffff9f2f7cc54000-0xffff9f2f7ffbffff] [ 4.366666] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.370598] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.375171] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.636881] Initialise system trusted keyrings [ 6.639892] Key type blacklist registered [ 6.644894] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.672201] zbud: loaded [ 6.682557] *** VALIDATE nfs *** [ 6.684892] *** VALIDATE nfs4 *** [ 6.694819] pstore: using deflate compression [ 6.712957] Platform Keyring initialized [ 7.094990] NET: Registered protocol family 38 [ 7.102085] Key type asymmetric registered [ 7.105136] Asymmetric key parser 'x509' registered [ 7.107867] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.114589] io scheduler mq-deadline registered [ 7.116701] io scheduler kyber registered [ 7.130365] io scheduler bfq registered [ 7.148602] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.154247] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.173474] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.176538] ACPI: Power Button [PWRF] [ 7.184705] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.202975] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.235082] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.245405] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.267552] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.306204] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.348900] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.353946] Non-volatile memory driver v1.3 [ 7.355706] Linux agpgart interface v0.103 [ 7.406924] virtio_blk virtio1: [vda] 149808 512-byte logical blocks (76.7 MB/73.1 MiB) [ 7.410539] vda: detected capacity change from 0 to 76701696 [ 7.431473] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.434761] vdb: detected capacity change from 0 to 1073741824 [ 7.452926] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.457032] vdc: detected capacity change from 0 to 2621440000 [ 7.480717] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.491063] vdd: detected capacity change from 0 to 2621440000 [ 7.515463] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.520713] vde: detected capacity change from 0 to 4294967296 [ 7.552506] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.556855] vdf: detected capacity change from 0 to 4294967296 [ 7.575288] libphy: Fixed MDIO Bus: probed [ 7.585785] usbcore: registered new interface driver usbserial_generic [ 7.592692] usbserial: USB Serial support registered for generic [ 7.598683] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.605933] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.608652] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.612424] mousedev: PS/2 mouse device common for all mice [ 7.616981] rtc_cmos 00:05: RTC can wake from S4 [ 7.620982] rtc_cmos 00:05: registered as rtc0 [ 7.622613] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.627129] intel_pstate: CPU model not supported [ 7.631420] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.634536] hid: raw HID events driver (C) Jiri Kosina [ 7.645635] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.647196] usbcore: registered new interface driver usbhid [ 7.647202] usbhid: USB HID core driver [ 7.648441] drop_monitor: Initializing network drop monitor service [ 7.654460] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.668852] Initializing XFRM netlink socket [ 7.678111] NET: Registered protocol family 10 [ 7.685706] Segment Routing with IPv6 [ 7.688590] NET: Registered protocol family 17 [ 7.692816] mpls_gso: MPLS GSO support [ 7.702068] RAS: Correctable Errors collector initialized. [ 7.707663] AVX version of gcm_enc/dec engaged. [ 7.713621] AES CTR mode by8 optimization enabled [ 7.951926] sched_clock: Marking stable (7951833832, 0)->(9003809472, -1051975640) [ 7.958731] registered taskstats version 1 [ 7.961481] Loading compiled-in X.509 certificates [ 7.964296] zswap: loaded using pool lzo/zbud [ 8.040265] Key type big_key registered [ 8.074661] Key type encrypted registered [ 8.076660] ima: No TPM chip found, activating TPM-bypass! [ 8.081836] ima: Allocated hash algorithm: sha1 [ 8.084217] ima: No architecture policies found [ 8.087816] evm: Initialising EVM extended attributes: [ 8.094092] evm: security.selinux [ 8.095699] evm: security.ima [ 8.097139] evm: security.capability [ 8.100215] evm: HMAC attrs: 0x1 [ 8.103897] rtc_cmos 00:05: setting system clock to 2026-09-06 05:27:43 UTC (1788672463) [ 8.115483] debug: unmapping init [mem 0xffffffff97e03000-0xffffffff97ffffff] [ 8.120775] debug: unmapping init [mem 0xffffffff96b82000-0xffffffff96e58fff] [ 8.129596] Write protecting the kernel read-only data: 28672k [ 8.133816] debug: unmapping init [mem 0xffffffff95203000-0xffffffff953fffff] [ 8.137490] debug: unmapping init [mem 0xffffffff95b14000-0xffffffff95bfffff] [ 8.206938] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.213084] systemd[1]: Detected virtualization kvm. [ 8.214502] systemd[1]: Detected architecture x86-64. [ 8.216124] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.256085] systemd[1]: No hostname configured. [ 8.259555] systemd[1]: Set hostname to . [ 8.262124] random: systemd: uninitialized urandom read (16 bytes read) [ 8.271530] systemd[1]: Initializing machine ID from random generator. [ 8.346071] random: ln: uninitialized urandom read (6 bytes read) [ 8.477890] random: systemd: uninitialized urandom read (16 bytes read) [ 8.480012] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 8.483731] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 8.487323] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.829136] device-mapper: uevent: version 1.0.3 [ 9.832403] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.113723] virtio_net virtio0 ens2: renamed from eth0 [ 11.123773] random: fast init done [ 11.557275] scsi host0: ata_piix [ 11.598380] scsi host1: ata_piix [ 11.600420] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.607151] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.616052] random: crng init done [ 16.618754] random: 7 urandom warning(s) missed due to ratelimiting [ 19.741913] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK [0[ 20.704440] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) m] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.396535] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.459297] SELinux: Disabled at runtime. [ 25.613788] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.655468] systemd[1]: Detected virtualization kvm. [ 25.662865] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.244625] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.255753] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.275719] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.287716] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.298279] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.317142] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.343439] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 27.971368] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 28.514947] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.284481] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.352579] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.782272] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.877704] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 34.543572] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (7s / no limit)[ 35.346147] NFS: Registering the id_resolver key type [ 35.349341] Key type id_resolver registered [ 35.354900] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg442-server login: [ 95.907175] libcfs: loading out-of-tree module taints kernel. [ 95.954763] Key type ._llcrypt registered [ 95.957740] Key type .llcrypt registered [ 96.092367] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 113.803812] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 114.933974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.955413] alg: No test for adler32 (adler32-zlib) [ 116.444515] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 117.229438] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 119.000976] Key type lgssc registered [ 120.635407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 135.976328] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 178.417548] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 192.955895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 193.006946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.353862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 194.406384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 194.533740] Lustre: lustre-MDT0000: new disk, initializing [ 194.633908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.656801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 199.663190] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.799110] hrtimer: interrupt took 5998913 ns [ 214.358484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.529478] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 214.600845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 214.608759] Lustre: Skipped 1 previous similar message [ 214.718343] Lustre: lustre-MDT0001: new disk, initializing [ 214.787381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.837567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 214.859963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 220.742843] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.037665] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 235.647286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.988662] Lustre: lustre-OST0000: new disk, initializing [ 235.997321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 236.018557] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 236.111788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.454491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 237.466068] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 237.535586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 242.494645] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.958856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.115900] Lustre: lustre-OST0001: new disk, initializing [ 258.120893] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 258.130295] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.210488] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 264.254608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 264.269515] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 264.390219] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 265.461105] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.079746] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 290.852120] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 298.092084] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing check_logdir /tmp/testlogs/ [ 303.802838] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing yml_node [ 309.205848] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 311.866811] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 314.835927] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 316.695481] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 01:32:50 EDT 2026 [ 334.149091] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 335.608808] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 351.711678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.715912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.725173] Lustre: Skipped 2 previous similar messages [ 351.739215] Lustre: Skipped 3 previous similar messages [ 356.832946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 356.849908] Lustre: Skipped 3 previous similar messages [ 357.965145] Lustre: server umount lustre-MDT0000 complete [ 362.222456] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672817 with bad export cookie 17090005469472148424 [ 362.232703] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.247242] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 367.072040] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.072696] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.088187] Lustre: Skipped 1 previous similar message [ 367.089715] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 367.140848] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 368.973943] Lustre: server umount lustre-MDT0001 complete [ 373.672610] Lustre: server umount lustre-OST0000 complete [ 377.939939] Lustre: server umount lustre-OST0001 complete [ 384.986298] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 395.705117] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 431.963122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.327764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 432.357062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 432.454252] Lustre: lustre-MDT0000: new disk, initializing [ 432.542926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.564663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 437.240552] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 448.390384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.591350] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 448.672094] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 448.679118] Lustre: Skipped 1 previous similar message [ 448.821594] Lustre: lustre-MDT0001: new disk, initializing [ 449.002819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 449.080146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 449.101556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 454.132131] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 464.159403] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 470.005552] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 476.581706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 476.885958] Lustre: lustre-OST0000: new disk, initializing [ 476.900983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 476.910704] Lustre: 17046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 477.003344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.624969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 478.643204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 478.796531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 483.158608] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 491.791473] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 495.608775] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 495.764110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 500.735237] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 500.993866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 504.311606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 504.324339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.354806] Lustre: Skipped 1 previous similar message [ 504.363479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 504.373635] Lustre: Skipped 2 previous similar messages [ 509.429656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 509.440859] Lustre: Skipped 1 previous similar message [ 510.031499] Lustre: server umount lustre-OST0000 complete [ 514.528672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.552334] Lustre: Skipped 1 previous similar message [ 519.649621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 519.660796] Lustre: Skipped 3 previous similar messages [ 520.822509] Lustre: server umount lustre-MDT0000 complete [ 524.331199] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672979 with bad export cookie 17090005469472149957 [ 524.343062] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 524.356528] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 524.790686] Lustre: server umount lustre-MDT0001 complete [ 530.328858] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:36:24 (1788672984) [ 550.270358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.733032] LustreError: 19921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.805718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.684442] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 556.008423] LustreError: 19922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.121615] LustreError: 19921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.224106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.411776] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 578.803450] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 585.436581] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 588.118725] Lustre: Failing over lustre-MDT0000 [ 588.291714] LustreError: 21489:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (ffff9f2fd9b61500) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 590.178513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.191312] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.201807] Lustre: Skipped 2 previous similar messages [ 590.241717] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 590.417976] Lustre: server umount lustre-MDT0000 complete [ 600.544638] LustreError: 20649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.564127] LustreError: 20649:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 606.369142] Lustre: 3645:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788673045/real 1788673045] req@ffff9f2ec7617100 x1875559131019264/t0(0) o400->MGC192.168.204.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788673061 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 606.409404] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 610.816873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 615.648166] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f2ec75c4a80 x1875559131022208/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 615.924285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 615.930670] Lustre: Skipped 1 previous similar message [ 615.989974] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 620.739823] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 621.043796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 621.043796] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 621.081532] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 631.238856] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 634.698280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.724975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 634.733276] Lustre: Skipped 1 previous similar message [ 640.994944] LustreError: 20649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.018977] LustreError: 20649:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 641.059091] Lustre: server umount lustre-MDT0000 complete [ 644.893882] LustreError: 19901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673100 with bad export cookie 17090005469472151371 [ 644.896315] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.913565] LustreError: 19901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 645.421746] Lustre: server umount lustre-MDT0001 complete [ 655.937458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.220109] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 672.218510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.631877] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 686.289722] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 693.001945] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 695.378131] Lustre: Failing over lustre-MDT0000 [ 695.802049] Lustre: server umount lustre-MDT0000 complete [ 697.314891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.334946] Lustre: Skipped 1 previous similar message [ 697.339235] LustreError: 23259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 697.356872] LustreError: 23259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 712.612142] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788673152/real 1788673152] req@ffff9f2fc6f90380 x1875559131049728/t0(0) o400->MGC192.168.204.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788673168 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 712.665566] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 714.965900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.945771] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xed2be56d815edaf5 [ 722.963264] Lustre: MGC192.168.204.142@tcp: Connection restored to 0@lo (at 0@lo) [ 722.981377] Lustre: Skipped 1 previous similar message [ 723.758961] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 729.056942] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 729.076270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 729.113922] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 730.800591] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 743.096709] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 749.537766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 749.541286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 749.545082] LustreError: Skipped 1 previous similar message [ 749.549888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 749.549896] Lustre: Skipped 2 previous similar messages [ 749.581193] Lustre: Skipped 2 previous similar messages [ 754.365795] Lustre: server umount lustre-MDT0000 complete [ 758.642749] LustreError: 23244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673214 with bad export cookie 17090005469472152309 [ 758.648237] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 758.661308] LustreError: 23244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 759.128988] Lustre: server umount lustre-MDT0001 complete [ 768.518648] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:40:22 (1788673222) [ 786.634505] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 800.192821] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 843.907558] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 855.504051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.714265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 855.783215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 855.901336] Lustre: lustre-MDT0000: new disk, initializing [ 855.981637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 855.992887] Lustre: Skipped 3 previous similar messages [ 856.002606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 861.246658] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 872.073737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.153380] Lustre: 29640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 872.387968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 878.257471] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 890.521693] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 895.957046] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 901.696626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 901.895703] Lustre: 31612:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 903.799892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 903.855753] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 903.925727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 903.940811] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 907.408703] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 918.124680] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 926.039022] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 945.123460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 945.123604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 945.132528] LustreError: Skipped 1 previous similar message [ 945.157225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 945.171995] Lustre: Skipped 2 previous similar messages [ 946.247383] Lustre: server umount lustre-OST0000 complete [ 956.561421] Lustre: server umount lustre-MDT0000 complete [ 958.436516] LustreError: 29651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.449328] LustreError: 29651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 960.534169] LustreError: 30436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673415 with bad export cookie 17090005469472152722 [ 960.539651] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.548644] LustreError: 30436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 960.914430] Lustre: server umount lustre-MDT0001 complete [ 970.691877] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 972.905491] Key type lgssc unregistered [ 973.301542] LNet: 33588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.326671] LNetError: 33588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.341045] LNet: Removed LNI 192.168.204.142@tcp [ 974.452573] Key type .llcrypt unregistered [ 974.454872] Key type ._llcrypt unregistered [ 1003.182512] Key type ._llcrypt registered [ 1003.184523] Key type .llcrypt registered [ 1003.274612] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1004.307071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1004.363246] alg: No test for adler32 (adler32-zlib) [ 1005.644467] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 1006.078510] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1007.823271] Key type lgssc registered [ 1009.462518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1017.551717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1017.564763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1019.246277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.536045] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1029.617919] LustreError: 34997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1029.662790] LustreError: 34997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1034.324691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1034.728196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1040.199262] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1047.975786] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1053.620863] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1060.867268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1061.159643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.347340] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1075.267533] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1080.631440] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1092.796953] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:45:46 (1788673546) [ 1094.513912] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1096.128918] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1102.310574] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:45:56 (1788673556) [ 1111.574396] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:46:05 (1788673565) [ 1112.530773] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1122.208754] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1122.216174] Lustre: Skipped 2 previous similar messages [ 1124.251667] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:46:17 (1788673577) [ 1126.367500] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1126.374554] Lustre: Skipped 2 previous similar messages [ 1135.308465] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1135.314534] Lustre: Skipped 4 previous similar messages [ 1150.674701] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:46:43 (1788673603) [ 1153.376156] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1156.496489] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1165.266627] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:46:58 (1788673618) [ 1167.696926] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1167.701722] Lustre: Skipped 6 previous similar messages [ 1183.330878] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:47:17 (1788673637) [ 1198.565107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1198.567674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.582119] LustreError: Skipped 1 previous similar message [ 1198.589536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1198.604073] Lustre: Skipped 1 previous similar message [ 1203.537830] Lustre: server umount lustre-OST0000 complete [ 1208.801412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1208.811473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.818787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1208.843084] Lustre: Skipped 1 previous similar message [ 1208.867862] Lustre: Skipped 2 previous similar messages [ 1213.849259] Lustre: server umount lustre-MDT0000 complete [ 1213.922697] LustreError: 34992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.951517] LustreError: 34992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1218.223251] LustreError: 34979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673673 with bad export cookie 122336356879070624 [ 1218.227296] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1218.235330] LustreError: 34979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1218.782969] Lustre: server umount lustre-MDT0001 complete [ 1230.485137] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1233.723754] Key type lgssc unregistered [ 1234.119746] LNet: 42852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1234.131959] LNetError: 42852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1234.160165] LNet: Removed LNI 192.168.204.142@tcp [ 1235.223609] Key type .llcrypt unregistered [ 1235.226520] Key type ._llcrypt unregistered [ 1252.468746] Key type ._llcrypt registered [ 1252.471183] Key type .llcrypt registered [ 1252.586622] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 1266.348984] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1267.360290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1267.374424] alg: No test for adler32 (adler32-zlib) [ 1268.536689] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 1268.783399] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1270.456644] Key type lgssc registered [ 1271.387120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1305.630882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1305.661454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1306.878140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1306.922147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1307.003767] Lustre: lustre-MDT0000: new disk, initializing [ 1307.082773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.098582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1311.416935] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1321.643488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1321.755645] Lustre: 46004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1321.812564] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1321.825624] Lustre: Skipped 1 previous similar message [ 1321.917707] Lustre: lustre-MDT0001: new disk, initializing [ 1322.083602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1322.121256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1322.132827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1326.470314] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1336.717587] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1343.308742] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1350.616968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1350.949175] Lustre: lustre-OST0000: new disk, initializing [ 1350.958368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1350.965990] Lustre: 47974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1351.092927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1357.472090] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1358.437060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1358.455075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1358.671449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1367.345861] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1372.751793] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1372.983830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1377.827241] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1378.050251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1384.420852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.439456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1384.444915] Lustre: Skipped 1 previous similar message [ 1384.454779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1384.458377] Lustre: Skipped 1 previous similar message [ 1386.199369] Lustre: server umount lustre-OST0000 complete [ 1393.632209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1393.637304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.648372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.656629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1394.664647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.687141] Lustre: Skipped 1 previous similar message [ 1399.780580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1399.792742] Lustre: Skipped 1 previous similar message [ 1403.873463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1403.882707] Lustre: Skipped 1 previous similar message [ 1404.383170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1404.621909] Lustre: server umount lustre-MDT0000 complete [ 1408.724659] LustreError: 47973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673864 with bad export cookie 5483070115518149582 [ 1408.734442] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1408.736077] LustreError: 47973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1409.060662] Lustre: server umount lustre-MDT0001 complete [ 1419.684363] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:51:13 (1788673873) [ 1426.685384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.413443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1432.545852] LustreError: 50435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1432.563397] LustreError: 50435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1432.885356] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1436.645369] LustreError: 50436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.587524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1441.763272] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1441.976259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1447.189326] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1455.714942] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1461.814305] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1468.997687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1469.426513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.649104] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1484.050189] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1491.797869] LustreError: 52963:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1492.723671] Lustre: Modifying parameter general.timeout=1 in log params [ 1494.267642] Lustre: Modifying parameter general.timeout=3 in log params [ 1494.272476] Lustre: Skipped 1 previous similar message [ 1496.897046] Lustre: Modifying parameter general.timeout=6 in log params [ 1496.901272] Lustre: Skipped 2 previous similar messages [ 1500.981758] Lustre: Modifying parameter general.timeout=11 in log params [ 1500.990292] Lustre: Skipped 4 previous similar messages [ 1509.786574] Lustre: Modifying parameter general.timeout=20 in log params [ 1509.792683] Lustre: Skipped 8 previous similar messages [ 1526.496334] Lustre: Modifying parameter general.timeout=37 in log params [ 1526.505707] Lustre: Skipped 16 previous similar messages [ 1559.305934] Lustre: Modifying parameter general.timeout=71 in log params [ 1559.309741] Lustre: Skipped 33 previous similar messages [ 1624.522732] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:54:38 (1788674078) [ 1625.516162] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1625.520608] Lustre: Skipped 59 previous similar messages [ 1627.386591] Lustre: 59906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1629.503606] Lustre: 60004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1633.079509] Lustre: 60215:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1633.097369] Lustre: 60215:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1637.568897] Lustre: 60421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1637.589525] Lustre: 60421:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1665.557808] Lustre: 61783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1665.611688] LustreError: 61783:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1669.391549] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:55:23 (1788674123) [ 1681.897297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.906134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1681.910417] Lustre: Skipped 2 previous similar messages [ 1687.017514] LustreError: 52276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1687.042572] LustreError: 52276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1687.162910] Lustre: server umount lustre-MDT0000 complete [ 1690.520859] LustreError: 50417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674145 with bad export cookie 5483070115518150464 [ 1690.523432] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1690.530804] LustreError: 50417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1690.868260] Lustre: server umount lustre-MDT0001 complete [ 1705.268076] Lustre: server umount lustre-OST0000 complete [ 1734.510642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.819412] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1735.155208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1740.378908] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1749.682842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1749.761759] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1749.794730] Lustre: 64316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1749.891552] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1754.995323] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1761.698342] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1767.072895] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1773.550476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1773.801823] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1774.187991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.200088] Lustre: Skipped 1 previous similar message [ 1780.813393] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1788.490900] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1790.330951] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1790.341588] Lustre: Skipped 22 previous similar messages [ 1790.419227] Lustre: 66886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1800.676385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1800.680553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.692476] Lustre: Skipped 2 previous similar messages [ 1803.264313] Lustre: server umount lustre-OST0000 complete [ 1810.914892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.932564] Lustre: Skipped 1 previous similar message [ 1814.761396] Lustre: server umount lustre-MDT0000 complete [ 1816.033269] LustreError: 64327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1816.055150] LustreError: 64327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1818.843683] LustreError: 66178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674274 with bad export cookie 5483070115518153439 [ 1818.844886] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1818.855193] LustreError: 66178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1819.418045] Lustre: server umount lustre-MDT0001 complete [ 1831.796236] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1834.402676] Key type lgssc unregistered [ 1834.695502] LNet: 68093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.709503] LNetError: 68093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1834.724755] LNet: Removed LNI 192.168.204.142@tcp [ 1835.482370] Key type .llcrypt unregistered [ 1835.488264] Key type ._llcrypt unregistered [ 1861.212679] Key type ._llcrypt registered [ 1861.214863] Key type .llcrypt registered [ 1861.343985] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1862.202922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1862.288724] alg: No test for adler32 (adler32-zlib) [ 1863.315295] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 1863.583960] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1865.207233] Key type lgssc registered [ 1866.265326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1874.181637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1874.191365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1875.702841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1879.923981] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1886.180278] LustreError: 69509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1886.188318] LustreError: 69509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1888.130317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.888078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1894.720067] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1901.783244] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1909.000963] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1915.712398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1916.324517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.482231] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1931.094740] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1939.969890] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1950.182344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1950.190173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1950.210969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.252819] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.254539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.277543] Lustre: Skipped 1 previous similar message [ 1953.284816] Lustre: Skipped 1 previous similar message [ 1955.023745] Lustre: server umount lustre-MDT0000 complete [ 1958.375464] LustreError: 70470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1958.385994] LustreError: 70470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1958.941696] LustreError: 69489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674414 with bad export cookie 1293078006537188366 [ 1958.961933] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1958.967123] LustreError: 69489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1959.350825] Lustre: server umount lustre-MDT0001 complete [ 1974.156807] Lustre: server umount lustre-OST0000 complete [ 1999.651869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1999.859201] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1999.876241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2000.048700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.967565] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2012.839057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.913144] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2012.937785] Lustre: 74377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2013.370990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2017.525249] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2024.593894] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2030.296211] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2036.418401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2036.617532] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2037.009643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.546555] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2048.073815] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2049.649538] Lustre: 76966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2051.277204] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2056.672831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2056.681816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.707631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2056.716255] Lustre: Skipped 1 previous similar message [ 2061.533902] Lustre: server umount lustre-OST0000 complete [ 2067.426581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.434564] Lustre: Skipped 1 previous similar message [ 2067.442077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2067.445392] Lustre: Skipped 2 previous similar messages [ 2071.472463] Lustre: server umount lustre-MDT0000 complete [ 2072.546150] LustreError: 74390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.575697] LustreError: 74390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2076.002700] LustreError: 74372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674531 with bad export cookie 1293078006537189185 [ 2076.007304] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2076.029386] LustreError: 74372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2076.532831] Lustre: server umount lustre-MDT0001 complete [ 2087.894107] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2090.067947] Key type lgssc unregistered [ 2090.392252] LNet: 78173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.408564] LNetError: 78173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.424781] LNet: Removed LNI 192.168.204.142@tcp [ 2091.330208] Key type .llcrypt unregistered [ 2091.336811] Key type ._llcrypt unregistered [ 2115.731941] Key type ._llcrypt registered [ 2115.734617] Key type .llcrypt registered [ 2115.846911] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2116.670150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2116.691449] alg: No test for adler32 (adler32-zlib) [ 2117.816754] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 2118.144582] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2119.815183] Key type lgssc registered [ 2120.813566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2128.625706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2128.670602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.304433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2135.306501] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2139.623748] LustreError: 79584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.646869] LustreError: 79584:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2144.074192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.791554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2148.961971] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2156.248945] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2161.221852] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2167.925693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2168.533078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.010369] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2182.340778] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2189.005798] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2189.940351] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2191.819994] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2191.825689] Lustre: Skipped 1 previous similar message [ 2193.869712] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2193.882982] Lustre: Skipped 1 previous similar message [ 2198.030407] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2198.041864] Lustre: Skipped 4 previous similar messages [ 2206.537854] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2206.543785] Lustre: Skipped 8 previous similar messages [ 2223.126350] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2223.132457] Lustre: Skipped 16 previous similar messages [ 2255.200178] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2255.215909] Lustre: Skipped 36 previous similar messages [ 2288.216521] Lustre: 87686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2290.843441] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 02:05:44 (1788674744) [ 2292.418550] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2294.537765] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 02:05:48 (1788674748) [ 2302.950591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.966172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2302.978556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2302.986846] Lustre: Skipped 1 previous similar message [ 2304.602218] Lustre: server umount lustre-OST0000 complete [ 2308.578269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2308.593659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.606315] Lustre: Skipped 1 previous similar message [ 2308.612696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2313.184735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2313.187847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2314.814676] Lustre: server umount lustre-MDT0000 complete [ 2318.305792] LustreError: 79585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.318431] LustreError: 79585:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2318.561093] LustreError: 79565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674773 with bad export cookie 10055081938504535223 [ 2318.571380] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2318.575371] LustreError: 79565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2318.842138] Lustre: server umount lustre-MDT0001 complete [ 2329.112683] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2331.763164] Key type lgssc unregistered [ 2331.995657] LNet: 88942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2332.000064] LNetError: 88942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2332.013516] LNet: Removed LNI 192.168.204.142@tcp [ 2332.941367] Key type .llcrypt unregistered [ 2332.946318] Key type ._llcrypt unregistered [ 2338.106820] Key type ._llcrypt registered [ 2338.108304] Key type .llcrypt registered [ 2338.226320] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2344.509323] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules [ 2346.051536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2346.327595] alg: No test for adler32 (adler32-zlib) [ 2347.810218] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 2348.324060] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2350.103240] Key type lgssc registered [ 2351.883379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2353.546184] LustreError: 90196:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2370.116430] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2374.877936] LustreError: 90196:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2378.757562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2378.782892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.288185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.544957] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2390.504577] LustreError: 91210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2390.528648] LustreError: 91210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2391.909607] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 02:07:26 (1788674846) [ 2395.616694] LustreError: 91211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.957833] LustreError: 91242:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2395.971463] LustreError: 91242:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2395.985445] LustreError: 91242:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2396.293369] Lustre: server umount lustre-MDT0000 complete [ 2411.186662] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2414.713812] Key type lgssc unregistered [ 2415.121579] LNet: 92734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.137424] LNetError: 92734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2416.175343] LNet: Removed LNI 192.168.204.142@tcp [ 2417.195454] Key type .llcrypt unregistered [ 2417.199635] Key type ._llcrypt unregistered [ 2436.875469] Key type ._llcrypt registered [ 2436.878292] Key type .llcrypt registered [ 2437.008305] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2438.438652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2438.587792] alg: No test for adler32 (adler32-zlib) [ 2439.611587] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 2439.810642] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2441.447329] Key type lgssc registered [ 2442.864119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2451.355647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2451.381966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.849050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2457.613146] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2463.201834] LustreError: 93950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.215680] LustreError: 93950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2465.614086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.191476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2470.131244] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2477.165277] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2482.872352] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2489.569682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2490.192818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2495.950619] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2502.687897] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2552.071797] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 02:10:05 (1788675005) [ 2554.819569] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2556.527872] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 02:10:10 (1788675010) [ 2563.041595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2563.071393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.108571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2565.096610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2565.109521] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2566.578832] Lustre: server umount lustre-MDT0000 complete [ 2570.217665] LustreError: 93946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.250671] LustreError: 93946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2570.579270] LustreError: 93930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675025 with bad export cookie 16241768218540364537 [ 2570.585927] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2570.592144] LustreError: 93930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2570.947831] Lustre: server umount lustre-MDT0001 complete [ 2585.596399] Lustre: server umount lustre-OST0000 complete [ 2597.064409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.428496] LustreError: 98244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.688067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2601.611425] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2602.981083] LustreError: 98245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.086149] LustreError: 98244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.964276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.072910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2617.260053] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2627.223318] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2632.625599] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2644.951136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2645.200244] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2645.204889] LustreError: 98232:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2645.219843] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2645.228925] LustreError: 100328:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2645.234512] LustreError: 100328:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2645.238281] LustreError: 100328:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2645.294971] Lustre: server umount lustre-OST0000 complete [ 2645.300980] LustreError: 100328:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2653.628586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2653.802164] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2653.817064] LustreError: 98233:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2653.856466] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2653.866300] LustreError: 100794:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2653.872353] LustreError: 100794:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2653.878140] LustreError: 100794:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2653.942511] Lustre: server umount lustre-OST0000 complete [ 2653.944302] LustreError: 100794:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2663.884879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2664.066572] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2664.070169] Lustre: Found index 0 for lustre-OST0000, updating log [ 2664.077951] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2664.163977] Lustre: lustre-OST0000: new disk, initializing [ 2664.170723] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2664.174909] Lustre: 101331:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2664.470624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.553918] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2674.777435] Lustre: Failing over lustre-OST0000 [ 2675.013749] Lustre: server umount lustre-OST0000 complete [ 2679.975489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.007561] Lustre: Skipped 1 previous similar message [ 2680.016761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2680.028426] Lustre: Skipped 3 previous similar messages [ 2685.411839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2685.428419] Lustre: Skipped 1 previous similar message [ 2689.570465] Lustre: server umount lustre-MDT0000 complete [ 2690.532504] LustreError: 99825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.549798] LustreError: 99825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2693.331308] LustreError: 98226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675148 with bad export cookie 16241768218540367281 [ 2693.342682] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2693.350209] LustreError: 98226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2693.735796] Lustre: server umount lustre-MDT0001 complete [ 2702.528923] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 02:12:36 (1788675156) [ 2717.769433] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2728.211642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.686308] LustreError: 103676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.855206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.687860] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2742.683918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2747.886600] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2750.776334] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2757.019595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.585509] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2763.111272] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2767.786241] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2767.793625] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2767.807168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2771.361924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2771.467701] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2771.534493] Lustre: lustre-OST0001: new disk, initializing [ 2771.538614] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2771.545705] Lustre: 105996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2771.809217] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 2771.840547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2771.857325] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2771.873524] Lustre: Skipped 2 previous similar messages [ 2773.941606] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2773.965921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2773.969123] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2773.972691] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2773.977728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2778.481262] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2780.258260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2780.274172] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2780.402169] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2786.495412] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2798.553279] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2804.708760] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2804.717812] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2804.734821] Lustre: Skipped 1 previous similar message [ 2804.743711] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2808.517266] Lustre: server umount lustre-MDT0001 complete [ 2809.893799] LustreError: 105191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.914171] LustreError: 105191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2819.509365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.592600] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2819.598336] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2819.650435] Lustre: 103666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2821.109380] LustreError: 107535:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2821.131050] LustreError: 107535:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2821.141152] LustreError: 103668:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2821.348877] LustreError: 107545:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.142@tcp: cfg command failed: rc = -17 [ 2821.374036] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2821.374036] [ 2825.715293] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2826.375162] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2835.193310] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 02:14:49 (1788675289) [ 2852.148816] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2868.177868] Lustre: 109613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2886.589637] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2889.546800] Lustre: 110747:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2968.545769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2968.560212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.578607] Lustre: Skipped 2 previous similar messages [ 2968.586945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2968.593879] Lustre: Skipped 4 previous similar messages [ 2973.209925] Lustre: server umount lustre-MDT0000 complete [ 2974.176540] LustreError: 103672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2974.196789] LustreError: 103672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 2976.583344] LustreError: 106715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675431 with bad export cookie 16241768218540368282 [ 2976.588342] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2976.597620] LustreError: 106715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2976.882621] Lustre: server umount lustre-MDT0001 complete [ 2990.625310] Lustre: server umount lustre-OST0000 complete [ 3004.447655] Lustre: server umount lustre-OST0001 complete [ 3009.773319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3018.132265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3024.669162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3033.247865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3052.142314] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3061.617320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3061.659612] Lustre: lustre-MDT0000: reset Object Index mappings [ 3062.282715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.295754] Lustre: Skipped 1 previous similar message [ 3066.621959] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3075.224503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.268558] Lustre: lustre-MDT0001: reset Object Index mappings [ 3081.299176] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3084.281450] Lustre: 115844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3090.474773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3096.138424] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3101.301094] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3103.427752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3106.996856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3107.015278] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3107.078861] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3109.186674] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3115.554683] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3118.834954] Lustre: 117720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3162.505632] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 02:20:16 (1788675616) [ 3193.825638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3193.849714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3193.863722] Lustre: Skipped 3 previous similar messages [ 3193.869793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3193.881493] Lustre: Skipped 4 previous similar messages [ 3196.084432] Lustre: server umount lustre-MDT0000 complete [ 3199.457880] LustreError: 115433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3199.477754] LustreError: 115433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3199.726032] LustreError: 114677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675655 with bad export cookie 16241768218540415357 [ 3199.739576] LustreError: 114677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3199.741166] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3214.303126] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3214.718448] Lustre: server umount lustre-MDT0001 complete [ 3219.045391] LustreError: 118482:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x16f:0x0].0x0 (ffff9f2ec9368700) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3225.327856] Lustre: server umount lustre-OST0000 complete [ 3229.184306] LustreError: 118683:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000401:0x3a:0x0].0x0 (ffff9f30007d8e00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3235.479869] Lustre: server umount lustre-OST0001 complete [ 3242.196687] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 3252.577235] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3286.541161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3286.773667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3286.824437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3286.910846] Lustre: lustre-MDT0000: new disk, initializing [ 3287.020190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3287.026556] Lustre: Skipped 3 previous similar messages [ 3287.046336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3291.728792] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3298.753641] Lustre: server umount lustre-MDT0000 complete [ 3308.124221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.540604] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3308.557465] Lustre: 121991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3308.574092] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3313.069505] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3318.189317] Lustre: server umount lustre-MDT0000 complete [ 3327.196394] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:23:00 (1788675780) [ 3328.829961] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3330.654749] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:23:04 (1788675784) [ 3347.968957] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3357.884653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.702312] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3370.752574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3370.876846] Lustre: 124039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3370.979169] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3370.982444] Lustre: Skipped 1 previous similar message [ 3371.059496] Lustre: lustre-MDT0001: new disk, initializing [ 3371.195065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3371.212426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3375.978807] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3381.074885] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3387.789329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3388.137933] Lustre: lustre-OST0000: new disk, initializing [ 3388.144522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3388.152059] Lustre: 125619:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3390.068675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3390.087875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3390.228759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3394.777550] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3404.781548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3404.941137] Lustre: lustre-OST0001: new disk, initializing [ 3404.944480] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3404.955703] Lustre: 126491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3406.847742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3406.855390] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3406.975491] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3410.646968] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3419.835344] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3423.181892] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3819.953854] Lustre: 132909:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3819.972085] Lustre: 132909:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3819.985517] Lustre: 132909:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3820.002223] Lustre: 132909:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3820.017794] Lustre: 132909:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3820.037773] Lustre: 132909:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4096.854957] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:35:49 (1788676549) [ 4113.390785] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4113.398170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4113.400798] Lustre: Skipped 7 previous similar messages [ 4113.440365] Lustre: Skipped 11 previous similar messages [ 4116.985691] Lustre: server umount lustre-MDT0000 complete [ 4118.504974] LustreError: 124046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4118.537760] LustreError: 124046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4120.608964] LustreError: 126490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676575 with bad export cookie 16241768218540476411 [ 4120.617125] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4120.632199] LustreError: 126490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4123.616262] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4123.619065] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4123.640779] Lustre: Skipped 2 previous similar messages [ 4123.652909] Lustre: Skipped 1 previous similar message [ 4127.345683] Lustre: server umount lustre-MDT0001 complete [ 4137.608035] Lustre: server umount lustre-OST0000 complete [ 4156.384993] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4156.724962] Lustre: server umount lustre-OST0001 complete [ 4163.881515] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 4175.511582] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4213.326994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4213.736106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4213.774801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4213.876286] Lustre: lustre-MDT0000: new disk, initializing [ 4214.000915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4214.005920] Lustre: Skipped 5 previous similar messages [ 4214.026903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4218.375861] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4229.861696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4230.069574] Lustre: 135963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4230.151790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4230.158355] Lustre: Skipped 1 previous similar message [ 4230.298156] Lustre: lustre-MDT0001: new disk, initializing [ 4230.456259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4230.479973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4235.584472] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4245.838484] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4251.849320] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4259.110820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4259.624640] Lustre: lustre-OST0000: new disk, initializing [ 4259.635152] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4259.642486] Lustre: 137928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4259.813835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4259.822440] Lustre: Skipped 1 previous similar message [ 4260.738517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4260.763650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4260.878595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4267.795948] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4278.357242] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4288.313485] Lustre: lustre-MDD0000: changelog on [ 4291.190383] Lustre: lustre-MDD0001: changelog on [ 6301.291627] Lustre: lustre-MDD0001: changelog off [ 6303.531895] Lustre: 135969:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6304.579755] Lustre: lustre-MDD0000: changelog off [ 6313.967734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.968301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6313.982975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6313.982984] Lustre: Skipped 2 previous similar messages [ 6317.873606] Lustre: server umount lustre-OST0000 complete [ 6324.194689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6324.197168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6324.206130] Lustre: Skipped 2 previous similar messages [ 6324.223586] Lustre: Skipped 2 previous similar messages [ 6328.551809] Lustre: server umount lustre-MDT0000 complete [ 6329.316330] LustreError: 135974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6329.348255] LustreError: 135974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6333.035850] LustreError: 135954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788678788 with bad export cookie 16241768218541256540 [ 6333.038028] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6333.047848] LustreError: 135954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6333.679607] Lustre: server umount lustre-MDT0001 complete [ 6346.427961] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6349.746955] Key type lgssc unregistered [ 6350.095679] LNet: 147095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6350.107769] LNetError: 147095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6350.126615] LNet: Removed LNI 192.168.204.142@tcp [ 6351.087310] Key type .llcrypt unregistered [ 6351.091383] Key type ._llcrypt unregistered [ 6375.101091] Key type ._llcrypt registered [ 6375.104991] Key type .llcrypt registered [ 6375.301967] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 6392.967140] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6394.177376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6394.299376] alg: No test for adler32 (adler32-zlib) [ 6395.446746] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 6395.817123] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6397.559199] Key type lgssc registered [ 6398.910297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6439.885785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6439.917247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6441.207414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6441.261670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6441.420085] Lustre: lustre-MDT0000: new disk, initializing [ 6441.533448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6441.563928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6447.648232] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6460.130758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.295429] Lustre: 150294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6460.370620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6460.377831] Lustre: Skipped 1 previous similar message [ 6460.491617] Lustre: lustre-MDT0001: new disk, initializing [ 6460.626424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6460.657779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6460.670189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6465.640323] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6476.608558] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6483.111372] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6491.536984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6491.947571] Lustre: lustre-OST0000: new disk, initializing [ 6491.958779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6491.969088] Lustre: 152264:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6492.155983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6499.622429] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6500.482885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6500.498416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6500.642778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6510.982705] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6518.517791] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_module obdecho/obdecho [ 6520.614271] LustreError: 153267:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6520.626927] LustreError: 153267:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6522.673642] LustreError: 153363:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6536.681113] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6536.698148] Lustre: Skipped 1 previous similar message [ 6536.700398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6536.715855] Lustre: Skipped 2 previous similar messages [ 6541.795925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6541.809441] Lustre: Skipped 2 previous similar messages [ 6542.155461] Lustre: server umount lustre-MDT0000 complete [ 6546.384795] LustreError: 150287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679001 with bad export cookie 4830357162620818144 [ 6546.392466] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6546.399352] LustreError: 150287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6546.912256] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6547.025325] Lustre: server umount lustre-MDT0001 complete [ 6553.373923] Lustre: server umount lustre-OST0000 complete [ 6575.600047] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 6588.049106] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6626.670201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.056630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6627.096751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6627.232545] Lustre: lustre-MDT0000: new disk, initializing [ 6627.320330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6627.355247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6632.221559] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6642.848974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6642.967172] Lustre: 157082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6643.008346] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6643.011379] Lustre: Skipped 1 previous similar message [ 6643.094420] Lustre: lustre-MDT0001: new disk, initializing [ 6643.170268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6643.206099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6643.221107] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6647.057601] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6657.013907] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6664.003243] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6670.965625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6671.318937] Lustre: lustre-OST0000: new disk, initializing [ 6671.324715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6671.330764] Lustre: 159053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6671.465509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6678.092559] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6678.639162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6678.653333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6678.878598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6689.045707] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6694.278878] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6694.533764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6699.739399] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6700.032297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6704.608703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6704.609849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6704.623610] LustreError: Skipped 1 previous similar message [ 6704.635505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6704.643323] Lustre: Skipped 1 previous similar message [ 6708.461875] Lustre: server umount lustre-OST0000 complete [ 6714.849509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6714.850736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6714.887243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6714.903472] Lustre: Skipped 2 previous similar messages [ 6718.642932] Lustre: server umount lustre-MDT0000 complete [ 6719.969805] LustreError: 157087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6719.989228] LustreError: 157087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6723.132418] LustreError: 157072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679178 with bad export cookie 4830357162620819103 [ 6723.134703] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6723.158720] LustreError: 157072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6723.767502] Lustre: server umount lustre-MDT0001 complete [ 6727.727518] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 03:19:41 (1788679181) [ 6729.786523] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6732.449532] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 03:19:45 (1788679185) [ 6742.578570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.056403] LustreError: 161466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6743.167647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6747.927257] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6748.646891] LustreError: 161467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6752.745238] LustreError: 161466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6756.622550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6762.036340] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6769.204244] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6774.506321] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6781.286094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6781.573160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6781.587161] Lustre: Skipped 1 previous similar message [ 6787.726888] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6795.213631] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6803.424837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6803.432059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6803.446042] Lustre: Skipped 1 previous similar message [ 6803.452173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6807.350209] Lustre: server umount lustre-MDT0000 complete [ 6809.571543] LustreError: 162194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6809.623638] LustreError: 162194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6811.777829] LustreError: 161448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679267 with bad export cookie 4830357162620819985 [ 6811.784667] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6811.784914] LustreError: 161448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6814.692244] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.723163] Lustre: Skipped 2 previous similar messages [ 6814.760344] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6814.768811] Lustre: Skipped 4 previous similar messages [ 6818.347844] Lustre: server umount lustre-MDT0001 complete [ 6828.713353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6843.599812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6844.004419] LustreError: 165310:0:(llog.c:1655:llog_backup()) MGC192.168.204.142@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6844.009944] Lustre: 165310:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6845.891208] LustreError: 165315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6845.918871] LustreError: 165315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6862.370987] LustreError: 165317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6862.391194] LustreError: 165317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6897.198892] LustreError: 165317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6897.215684] LustreError: 165317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 6904.802780] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6904.812433] Lustre: 165310:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.142@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6945.787753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6949.980313] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6950.888063] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6957.709946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6962.160511] LustreError: 165317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6962.195721] LustreError: 165317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 6972.387745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6987.531977] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6988.262114] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6994.804723] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6996.361449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7002.940986] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7004.641640] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7027.508236] Lustre: server umount lustre-MDT0000 complete [ 7029.221157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7029.225157] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7029.239273] Lustre: Skipped 1 previous similar message [ 7031.613362] LustreError: 164249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679487 with bad export cookie 4830357162620826474 [ 7031.619394] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7031.634942] LustreError: 164249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7032.020501] Lustre: server umount lustre-MDT0001 complete [ 7046.742130] Lustre: server umount lustre-OST0000 complete [ 7070.490971] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 7084.139784] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7122.942673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7123.257814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7123.300782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7123.437173] Lustre: lustre-MDT0000: new disk, initializing [ 7123.556589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7123.569208] Lustre: Skipped 1 previous similar message [ 7123.612778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7129.325091] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7140.560045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7140.744324] Lustre: 170419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7141.089780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7145.808717] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7155.721896] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7162.605886] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7170.246460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7170.600892] Lustre: lustre-OST0000: new disk, initializing [ 7170.607729] Lustre: Skipped 1 previous similar message [ 7170.612610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7170.621834] Lustre: Skipped 2 previous similar messages [ 7170.628427] Lustre: 172389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7176.766525] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7180.373795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7180.390346] Lustre: Skipped 1 previous similar message [ 7180.396818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7180.495647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7187.589989] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7193.255099] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7193.507667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7200.557887] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7200.820358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7206.370383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7206.376796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7206.388956] Lustre: Skipped 1 previous similar message [ 7206.396676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7206.396694] Lustre: Skipped 2 previous similar messages [ 7209.451938] Lustre: server umount lustre-OST0000 complete [ 7220.124351] Lustre: server umount lustre-MDT0000 complete [ 7221.732652] LustreError: 170426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7221.755271] LustreError: 170426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 21 previous similar messages [ 7224.725392] LustreError: 170410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679680 with bad export cookie 4830357162620843281 [ 7224.729181] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7224.742828] LustreError: 170410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7225.454137] Lustre: server umount lustre-MDT0001 complete [ 7235.543531] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:28:08 (1788679688) [ 7244.165872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7249.575508] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7258.704497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7264.622922] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7272.698772] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7278.352623] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7286.303062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7294.728482] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7303.630375] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7313.392294] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:29:27 (1788679767) [ 7319.522762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7319.524945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7319.531553] Lustre: Skipped 2 previous similar messages [ 7319.534370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7319.540738] LustreError: Skipped 1 previous similar message [ 7319.550453] Lustre: Skipped 4 previous similar messages [ 7325.140764] Lustre: server umount lustre-OST0000 complete [ 7335.991576] Lustre: server umount lustre-MDT0000 complete [ 7339.978368] LustreError: 174781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679795 with bad export cookie 4830357162620844156 [ 7339.988733] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7339.993864] LustreError: 174781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7340.010765] LustreError: 148518:0:(import.c:761:ptlrpc_connect_import_locked()) already connecting [ 7340.290190] Lustre: server umount lustre-MDT0001 complete [ 7350.874635] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 7354.297252] Key type lgssc unregistered [ 7354.585669] LNet: 178529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7354.601494] LNetError: 178529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7354.618383] LNet: Removed LNI 192.168.204.142@tcp [ 7355.598811] Key type .llcrypt unregistered [ 7355.601365] Key type ._llcrypt unregistered [ 7376.395429] Key type ._llcrypt registered [ 7376.397238] Key type .llcrypt registered [ 7376.483763] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7377.161585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7377.188213] alg: No test for adler32 (adler32-zlib) [ 7378.214022] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 7378.515587] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 7380.159793] Key type lgssc registered [ 7381.462623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7391.874453] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7391.909077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7407.712409] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7412.831642] LustreError: 179768:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 7417.951951] LustreError: 179768:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7417.975524] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7417.986203] LustreError: MGC192.168.204.142@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7417.999308] LustreError: 179768:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7418.007867] LustreError: 179768:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 7418.014480] LustreError: 179768:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7418.130166] Lustre: server umount lustre-OST0000 complete [ 7418.135508] LustreError: 179768:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7429.885868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7431.223650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7436.099201] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7441.382926] LustreError: 180270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7441.398904] LustreError: 180270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7444.474044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7444.932600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7449.697509] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7457.320672] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7463.517874] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7471.792802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7472.066207] LustreError: 182086:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7472.090396] Lustre: 182086:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.142@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7472.322788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7480.221737] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7486.894532] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7489.639732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7489.644292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7489.653085] Lustre: Skipped 1 previous similar message [ 7489.679826] Lustre: Skipped 2 previous similar messages [ 7495.001898] Lustre: server umount lustre-MDT0000 complete [ 7495.137487] LustreError: 180263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7495.179120] LustreError: 180263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7499.369633] LustreError: 180249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679954 with bad export cookie 12836574429934015483 [ 7499.376955] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7499.380286] LustreError: 180249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7499.959614] LustreError: 181000:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d59f48bc x1875566744596992/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7500.197957] Lustre: server umount lustre-MDT0001 complete [ 7514.889853] Lustre: server umount lustre-OST0000 complete [ 7524.582911] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 7536.603261] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7574.622623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7574.874386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7574.917826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7575.021154] Lustre: lustre-MDT0000: new disk, initializing [ 7575.094360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7575.117523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7580.275465] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7591.260698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7591.414525] Lustre: 185614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7591.459238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7591.464094] Lustre: Skipped 1 previous similar message [ 7591.566988] Lustre: lustre-MDT0001: new disk, initializing [ 7591.706709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7591.752950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7591.762666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7596.780165] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7606.193095] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7611.455149] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7618.119539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7618.455402] Lustre: lustre-OST0000: new disk, initializing [ 7618.460881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7618.467652] Lustre: 187580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7618.542830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7625.222819] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7625.268469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7625.285441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7625.366257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7633.880404] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7638.180482] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7638.350895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7642.637273] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7642.846410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7646.179351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7646.180858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7646.193567] LustreError: Skipped 1 previous similar message [ 7646.202362] Lustre: Skipped 1 previous similar message [ 7646.206422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7646.223668] Lustre: Skipped 1 previous similar message [ 7651.162901] Lustre: server umount lustre-OST0000 complete [ 7656.418600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7656.436822] Lustre: Skipped 1 previous similar message [ 7656.442862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7656.453929] Lustre: Skipped 1 previous similar message [ 7660.955848] Lustre: server umount lustre-MDT0000 complete [ 7661.537122] LustreError: 185620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7661.567813] LustreError: 185620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7665.273370] LustreError: 185605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788680120 with bad export cookie 12836574429934016204 [ 7665.276122] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7665.288478] LustreError: 185605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7665.855520] Lustre: server umount lustre-MDT0001 complete [ 7676.962662] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:35:30 (1788680130) [ 7696.157891] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 7699.636282] Key type lgssc unregistered [ 7700.051122] LNet: 190523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7700.061558] LNetError: 190523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7700.091639] LNet: Removed LNI 192.168.204.142@tcp [ 7700.957763] Key type .llcrypt unregistered [ 7700.960144] Key type ._llcrypt unregistered [ 7721.320489] Key type ._llcrypt registered [ 7721.323258] Key type .llcrypt registered [ 7721.413229] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7722.552684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7722.591374] alg: No test for adler32 (adler32-zlib) [ 7723.755830] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 7723.945130] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 7725.599193] Key type lgssc registered [ 7727.420229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7738.253662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7738.271571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7740.226763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7746.593699] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7750.628065] LustreError: 191742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7750.660961] LustreError: 191742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7755.745769] LustreError: 191743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7756.490769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7756.898208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7762.290084] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7769.946373] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7776.010709] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7784.019883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7784.222140] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7784.227501] LustreError: 193607:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7784.235815] Lustre: 193607:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.142@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7784.409296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7790.635935] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7798.027689] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7801.826466] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7801.834929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7801.849693] Lustre: Skipped 1 previous similar message [ 7801.852402] Lustre: Skipped 2 previous similar messages [ 7802.923907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7802.941837] Lustre: Skipped 3 previous similar messages [ 7806.679085] Lustre: server umount lustre-MDT0000 complete [ 7807.969162] LustreError: 191738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7807.995413] LustreError: 191738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7810.229616] LustreError: 191724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788680265 with bad export cookie 1160315701035914570 [ 7810.244442] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7810.250357] LustreError: 191724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7811.168459] LustreError: 192472:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008c63c7c9 x1875567107404160/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7811.329233] Lustre: server umount lustre-MDT0001 complete [ 7825.538168] Lustre: server umount lustre-OST0000 complete [ 7835.962117] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 7848.072470] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7886.722278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7886.955932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7886.998943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7887.082859] Lustre: lustre-MDT0000: new disk, initializing [ 7887.170101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7887.195279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7891.736106] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7902.529368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7902.691490] Lustre: 197133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7902.762583] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7902.764478] Lustre: Skipped 1 previous similar message [ 7902.768369] LustreError: 197915:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7902.774220] LustreError: 197915:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7902.780261] Lustre: 197915:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7902.797950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7902.802484] Lustre: Skipped 1 previous similar message [ 7902.906754] Lustre: lustre-MDT0001: new disk, initializing [ 7903.034975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7903.068924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7903.084499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7908.200158] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7918.472473] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7924.567891] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7930.964446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7931.221793] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7931.224020] Lustre: Skipped 1 previous similar message [ 7931.227643] LustreError: 199078:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7931.236093] LustreError: 199078:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7931.243643] Lustre: 199078:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7931.252296] Lustre: 199078:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7931.289918] Lustre: lustre-OST0000: new disk, initializing [ 7931.293139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7931.299733] Lustre: 199102:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7931.405754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7938.070857] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7938.662266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7938.684623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7938.839253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7948.737128] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7954.939750] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7955.228975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7960.878114] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7961.091664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7964.643178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7964.650291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7964.656120] LustreError: Skipped 1 previous similar message [ 7964.657118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7964.662931] Lustre: Skipped 1 previous similar message [ 7969.410692] Lustre: server umount lustre-OST0000 complete [ 7974.884290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7974.885452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7974.909674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7974.931895] Lustre: Skipped 2 previous similar messages [ 7980.006171] LustreError: 197139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7980.069305] Lustre: server umount lustre-MDT0000 complete [ 7983.832423] LustreError: 197126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788680439 with bad export cookie 1160315701035915291 [ 7983.836762] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7983.843932] LustreError: 197126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7984.172747] Lustre: server umount lustre-MDT0001 complete [ 7993.484456] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:40:46 (1788680446) [ 8012.225373] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8024.487798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8024.842872] LustreError: 202233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8024.877204] LustreError: 202233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8024.929642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8030.489314] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8035.297394] LustreError: 202234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8035.325128] LustreError: 202234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8039.947569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8046.295716] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8049.798107] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8057.457450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8057.922733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8057.931284] Lustre: Skipped 1 previous similar message [ 8065.066230] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8074.688513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8074.916540] Lustre: lustre-OST0001: new disk, initializing [ 8074.920924] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8074.930127] Lustre: 204543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8081.064770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8081.085131] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8081.292262] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8081.639770] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8093.101067] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8100.847761] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8103.342472] Lustre: DEBUG MARKER: ADD OST3 [ 8109.553295] loop: module loaded [ 8109.676915] loop0: detected capacity change from 0 to 204800000 [ 8110.014537] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8110.028295] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8110.044191] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8117.258736] loop0: detected capacity change from 0 to 204800000 [ 8117.551678] Lustre: DEBUG MARKER: STOP OST3 [ 8123.883896] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8124.053963] Lustre: lustre-OST0003: new disk, initializing [ 8124.056587] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8124.071539] Lustre: 206586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8124.258941] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8124.270709] Lustre: Skipped 1 previous similar message [ 8125.699079] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8125.838638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8125.865712] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8125.910923] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8127.507786] Lustre: Failing over lustre-OST0003 [ 8127.711865] LustreError: 206708:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8127.960851] Lustre: server umount lustre-OST0003 complete [ 8131.039935] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8131.060915] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8131.070292] LustreError: 203720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8131.080433] Lustre: Skipped 2 previous similar messages [ 8131.110591] LustreError: 203720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8132.358488] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8142.406292] Lustre: DEBUG MARKER: START OST3 again [ 8150.077606] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8150.556322] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8151.591997] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8152.321793] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8152.328272] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8156.984691] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8178.250697] Lustre: Failing over lustre-OST0003 [ 8178.346131] Lustre: server umount lustre-OST0003 complete [ 8181.238284] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8181.247630] LustreError: 208659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8181.266154] Lustre: Skipped 1 previous similar message [ 8181.298422] LustreError: 208659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8199.141879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8199.152370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8199.165893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8203.438501] Lustre: server umount lustre-MDT0000 complete [ 8207.871929] LustreError: 202214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788680663 with bad export cookie 1160315701035916173 [ 8207.875380] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8207.894580] LustreError: 202214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8211.952889] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8211.963544] Lustre: Skipped 5 previous similar messages [ 8214.616307] Lustre: server umount lustre-MDT0001 complete [ 8219.301138] Lustre: server umount lustre-OST0000 complete [ 8223.561496] Lustre: server umount lustre-OST0001 complete [ 8231.314569] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8240.520229] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8282.258704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8282.576472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8282.600455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8282.682049] Lustre: lustre-MDT0000: new disk, initializing [ 8282.787950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8282.795168] Lustre: Skipped 1 previous similar message [ 8282.813747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8287.573555] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8298.877581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8299.021670] Lustre: 211995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8299.081955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8299.086299] Lustre: Skipped 1 previous similar message [ 8299.189451] Lustre: lustre-MDT0001: new disk, initializing [ 8299.358565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8299.365159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8304.976634] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8315.500915] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8321.862853] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8329.074432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8329.353894] Lustre: 213966:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8331.037964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8331.197266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8337.234910] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8348.896221] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8354.775352] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8355.068310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8360.550669] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8360.833412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8367.078205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8367.080224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8367.092138] LustreError: Skipped 1 previous similar message [ 8367.098438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8367.111684] Lustre: Skipped 6 previous similar messages [ 8367.115558] Lustre: Skipped 2 previous similar messages [ 8377.311617] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8377.546412] Lustre: server umount lustre-OST0000 complete [ 8396.255191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8396.472191] Lustre: server umount lustre-MDT0000 complete [ 8397.797588] LustreError: 212009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8397.812000] LustreError: 212009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 8401.156404] LustreError: 211988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788680856 with bad export cookie 1160315701035918469 [ 8401.161532] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8401.168676] LustreError: 211988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8401.797134] Lustre: server umount lustre-MDT0001 complete [ 8406.768458] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:47:39 (1788680859) [ 8425.825170] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8438.670153] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8480.041621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8480.392735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8480.453124] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8480.459504] Lustre: Skipped 1 previous similar message [ 8480.521403] Lustre: lustre-MDT0000: new disk, initializing [ 8480.524474] Lustre: Skipped 1 previous similar message [ 8480.667333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8480.679322] Lustre: Skipped 1 previous similar message [ 8486.435653] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8497.886763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8498.077849] Lustre: 218583:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8498.435557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8503.031728] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8512.408180] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8518.481238] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8526.479970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8526.826130] Lustre: 220552:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8528.618119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8535.009233] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8545.959727] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8551.002778] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8551.255132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8557.105085] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8557.376268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8559.584112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8559.597095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8559.603345] LustreError: Skipped 1 previous similar message [ 8559.606044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8559.606053] Lustre: Skipped 11 previous similar messages [ 8559.653074] Lustre: Skipped 3 previous similar messages [ 8565.607587] Lustre: server umount lustre-OST0000 complete [ 8575.768485] Lustre: server umount lustre-MDT0000 complete [ 8579.825520] LustreError: 218575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788681035 with bad export cookie 1160315701035919351 [ 8579.843650] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8579.845448] LustreError: 218575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8580.253823] Lustre: server umount lustre-MDT0001 complete [ 8589.151201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8589.692526] LustreError: 222767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8589.787990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8589.796947] Lustre: Skipped 5 previous similar messages [ 8594.555348] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8602.931147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8607.221801] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8613.942924] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8619.195269] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8627.166715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8633.330271] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8641.503898] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8654.314792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8654.323639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8654.344091] Lustre: Skipped 2 previous similar messages [ 8654.362856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8654.380566] Lustre: Skipped 7 previous similar messages [ 8657.484536] Lustre: server umount lustre-OST0000 complete [ 8676.319191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8676.594123] Lustre: server umount lustre-MDT0000 complete [ 8680.889735] LustreError: 222748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788681136 with bad export cookie 1160315701035920233 [ 8680.903023] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8680.909332] LustreError: 222748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8681.254660] Lustre: server umount lustre-MDT0001 complete [ 8782.789554] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 8786.093484] Key type lgssc unregistered [ 8786.461610] LNet: 226444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8786.477167] LNetError: 226444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8786.494969] LNet: Removed LNI 192.168.204.142@tcp [ 8787.556169] Key type .llcrypt unregistered [ 8787.560245] Key type ._llcrypt unregistered [ 8817.488858] Key type ._llcrypt registered [ 8817.494348] Key type .llcrypt registered [ 8817.639815] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8836.172581] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8837.314672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8837.369529] alg: No test for adler32 (adler32-zlib) [ 8838.445768] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 8838.730304] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 8840.431159] Key type lgssc registered [ 8841.757565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8879.160965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8879.182529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8880.426980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8880.474652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8880.601867] Lustre: lustre-MDT0000: new disk, initializing [ 8880.748506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8880.770192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8886.098538] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8897.314872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8897.519736] Lustre: 229797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8897.611801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8897.615874] Lustre: Skipped 1 previous similar message [ 8897.687544] Lustre: lustre-MDT0001: new disk, initializing [ 8897.761857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8897.818198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8897.836832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8902.864498] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8913.995396] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8919.825508] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8927.136347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8927.428732] Lustre: lustre-OST0000: new disk, initializing [ 8927.438942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8927.452505] Lustre: 231767:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8927.539934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8933.955080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8933.972686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8934.087171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8934.225169] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8944.289186] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8948.890712] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8949.090117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8954.296439] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8954.615290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8959.970946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8959.972614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8960.018973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8960.028723] Lustre: Skipped 1 previous similar message [ 8962.725576] Lustre: server umount lustre-OST0000 complete [ 8969.696419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8969.711581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8969.732326] Lustre: Skipped 1 previous similar message [ 8969.741333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8972.925385] Lustre: server umount lustre-MDT0000 complete [ 8976.691407] LustreError: 229788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788681432 with bad export cookie 10022838183475960585 [ 8976.699093] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8976.702780] LustreError: 229788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8977.263291] Lustre: server umount lustre-MDT0001 complete [ 8997.863511] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9008.875226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.226390] LustreError: 234701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9009.250674] LustreError: 234701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9009.300167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9014.137775] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9014.753564] LustreError: 234702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9018.851417] LustreError: 234701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9022.837390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9023.223907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9027.528579] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9030.088531] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9036.853115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9037.235815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9043.771125] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9052.421740] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9052.605580] Lustre: lustre-OST0001: new disk, initializing [ 9052.608711] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9052.614906] Lustre: 237008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9055.037688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9055.043641] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9055.177286] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9058.905162] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9070.876168] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9081.211553] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9093.660712] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:59:07 (1788681547) [ 9096.167061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9096.173198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9096.190183] Lustre: Skipped 4 previous similar messages [ 9096.199148] Lustre: Skipped 4 previous similar messages [ 9101.283821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9101.290130] Lustre: Skipped 3 previous similar messages [ 9101.701812] Lustre: server umount lustre-MDT0000 complete [ 9105.727365] LustreError: 237836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788681561 with bad export cookie 10022838183475961467 [ 9105.746686] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9105.754936] LustreError: 237836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9106.110768] Lustre: server umount lustre-MDT0001 complete [ 9120.120782] Lustre: server umount lustre-OST0000 complete [ 9122.273587] Lustre: 228018:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788681561/real 1788681561] req@ffff9f2ed0ef8e00 x1875568276653696/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788681577 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9122.295784] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9123.645977] Lustre: server umount lustre-OST0001 complete [ 9130.029679] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9139.129379] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9172.160653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9172.328687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9172.352197] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9172.419148] Lustre: lustre-MDT0000: new disk, initializing [ 9172.484147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9172.487558] Lustre: Skipped 1 previous similar message [ 9172.503490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9176.495489] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9185.673663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9185.805401] Lustre: 240954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9185.871936] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9185.875071] Lustre: Skipped 1 previous similar message [ 9186.018843] Lustre: lustre-MDT0001: new disk, initializing [ 9186.214202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9186.230196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9190.610252] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9199.019453] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9203.858753] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9210.775578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9211.013457] Lustre: lustre-OST0000: new disk, initializing [ 9211.017179] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9211.023564] Lustre: 242931:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9211.109947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9211.114851] Lustre: Skipped 1 previous similar message [ 9212.897608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9212.910634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9213.053302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9217.608328] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9227.577512] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9233.121361] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9233.349518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9238.178096] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9238.462724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9243.619358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9243.626874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9243.643842] LustreError: Skipped 1 previous similar message [ 9243.655429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9243.674393] Lustre: Skipped 1 previous similar message [ 9243.722831] Lustre: Skipped 1 previous similar message [ 9246.739625] Lustre: server umount lustre-OST0000 complete [ 9252.833789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9252.843487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9252.856494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9256.256150] Lustre: server umount lustre-MDT0000 complete [ 9258.978551] LustreError: 240960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9259.000893] LustreError: 240960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9260.056924] LustreError: 240947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788681715 with bad export cookie 10022838183475963602 [ 9260.067033] LustreError: 240947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9260.067808] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9260.558995] Lustre: server umount lustre-MDT0001 complete [ 9268.695256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9269.150303] LustreError: 245147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9274.192494] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9279.467435] LustreError: 245148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9279.490458] LustreError: 245148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9283.310491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9283.841316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9283.855308] Lustre: Skipped 1 previous similar message [ 9288.929682] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9296.774855] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9302.170667] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9309.146412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9315.764420] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9323.169525] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9472.481760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9472.483673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9472.507040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9472.507049] Lustre: Skipped 2 previous similar messages [ 9472.538165] Lustre: Skipped 2 previous similar messages [ 9475.885520] Lustre: server umount lustre-OST0000 complete [ 9485.940970] Lustre: server umount lustre-MDT0000 complete [ 9487.844235] LustreError: 245148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9487.867161] LustreError: 245148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9489.840102] LustreError: 245127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788681945 with bad export cookie 10022838183475964477 [ 9489.852439] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9489.855480] LustreError: 245127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9490.271171] Lustre: server umount lustre-MDT0001 complete [ 9594.802821] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 9598.089423] Key type lgssc unregistered [ 9598.424549] LNet: 248754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9598.433371] LNetError: 248754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9598.455425] LNet: Removed LNI 192.168.204.142@tcp [ 9599.293390] Key type .llcrypt unregistered [ 9599.297668] Key type ._llcrypt unregistered [ 9631.986718] Key type ._llcrypt registered [ 9631.988860] Key type .llcrypt registered [ 9632.130359] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9649.286941] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9650.959640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9651.030110] alg: No test for adler32 (adler32-zlib) [ 9652.133634] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 9652.414458] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 9654.119167] Key type lgssc registered [ 9655.281080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9707.244035] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9720.285082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9720.306801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9721.598822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9721.681498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9721.818626] Lustre: lustre-MDT0000: new disk, initializing [ 9721.917842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9721.943752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9726.732928] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9736.900392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9737.078816] Lustre: 252826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9737.144177] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9737.149605] Lustre: Skipped 1 previous similar message [ 9737.221418] Lustre: lustre-MDT0001: new disk, initializing [ 9737.302654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9737.361645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9737.373116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9741.698331] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9746.483987] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9753.408092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9753.597208] Lustre: lustre-OST0000: new disk, initializing [ 9753.601436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9753.617990] Lustre: 254456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9753.764244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9755.663887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9755.683971] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9755.857913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9760.478622] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9771.666866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9771.887192] Lustre: lustre-OST0001: new disk, initializing [ 9771.892687] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9771.899548] Lustre: 255327:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9772.004737] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9779.307624] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9782.378813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9782.386951] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9782.450283] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9790.674585] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9799.608647] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9813.206285] Lustre: server umount lustre-MDT0000 complete [ 9813.474361] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9816.878430] LustreError: 252817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788682272 with bad export cookie 17903311271415022131 [ 9816.881409] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9816.898689] LustreError: 252817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9817.686959] Lustre: server umount lustre-MDT0001 complete [ 9832.017705] Lustre: server umount lustre-OST0000 complete [ 9833.823404] Lustre: 250328:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788682273/real 1788682273] req@ffff9f2fc35a2300 x1875569129105792/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788682289 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9833.866077] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9833.878191] Lustre: Skipped 3 previous similar messages [ 9836.496593] Lustre: server umount lustre-OST0001 complete [ 9849.798405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9859.576138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9867.525318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9876.306635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9895.292798] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9905.183926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9905.204137] Lustre: lustre-MDT0000: reset Object Index mappings [ 9905.627092] LustreError: 260340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9905.654019] LustreError: 260340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9905.752952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9909.965685] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9910.756874] LustreError: 260341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9915.885334] LustreError: 260340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9919.240949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9919.264476] Lustre: lustre-MDT0001: reset Object Index mappings [ 9919.710249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9924.870363] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9928.092229] Lustre: 261484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9935.133757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9941.904667] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9943.663315] LustreError: 261838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9949.159592] LustreError: 262285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9951.229550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9951.432394] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9951.677357] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9951.685676] Lustre: Skipped 1 previous similar message [ 9956.880611] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9958.788167] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9966.261811] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9970.104701] Lustre: 263355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9975.777870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9975.786615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9975.817588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9977.083182] Lustre: server umount lustre-MDT0000 complete [ 9977.321241] LustreError: 260341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9977.355672] LustreError: 260341:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9980.710111] LustreError: 263357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788682436 with bad export cookie 17903311271415028921 [ 9980.714601] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9980.725423] LustreError: 263357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9981.097270] Lustre: server umount lustre-MDT0001 complete [ 9995.762904] Lustre: server umount lustre-OST0000 complete [ 9998.559509] Lustre: 250330:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788682437/real 1788682437] req@ffff9f2fe5e24e00 x1875569129153408/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788682453 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9998.584624] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9998.596357] Lustre: Skipped 3 previous similar messages [10000.083924] Lustre: server umount lustre-OST0001 complete [10020.174491] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 04:14:33 (1788682473) [10035.036567] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [10044.421245] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10079.243470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10079.491225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10079.547573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10079.656876] Lustre: lustre-MDT0000: new disk, initializing [10079.755145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10079.782792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10084.101779] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10094.593693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10094.688617] Lustre: 267607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10094.733368] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10094.736552] Lustre: Skipped 1 previous similar message [10094.795334] Lustre: lustre-MDT0001: new disk, initializing [10094.920334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10094.930324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10099.723697] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10109.035812] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10113.693959] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10120.142992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10120.499501] Lustre: lustre-OST0000: new disk, initializing [10120.505683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10120.515302] Lustre: 269577:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10122.359878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10122.367119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10122.498318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10127.214981] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10136.686805] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10142.317038] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10142.504633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10147.283311] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10147.536841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10153.441150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10153.441599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10153.459473] LustreError: Skipped 1 previous similar message [10153.475651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10153.487529] Lustre: Skipped 1 previous similar message [10153.540358] Lustre: Skipped 1 previous similar message [10155.978891] Lustre: server umount lustre-OST0000 complete [10161.632334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10161.637097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10161.666883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10163.682448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10166.076913] Lustre: server umount lustre-MDT0000 complete [10168.800367] LustreError: 267613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10168.810448] LustreError: 267613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10169.645122] LustreError: 269576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788682625 with bad export cookie 17903311271415031014 [10169.656717] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10169.663806] LustreError: 269576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10170.017615] Lustre: server umount lustre-MDT0001 complete [10189.056123] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10197.918422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10198.343775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10198.355162] Lustre: Skipped 2 previous similar messages [10203.621783] LustreError: 272512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10203.677613] LustreError: 272512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10204.645980] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10214.235328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10219.947476] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10223.264923] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10229.723481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10236.748514] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10246.028708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10246.309533] Lustre: lustre-OST0001: new disk, initializing [10246.317257] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10246.322847] Lustre: 274818:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10247.525817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10247.535597] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10247.649480] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10252.961336] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10262.565318] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10265.966444] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10271.060329] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10271.385314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10281.441473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10281.454412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10281.463641] Lustre: Skipped 1 previous similar message [10281.468707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10281.471302] Lustre: Skipped 1 previous similar message [10282.282673] Lustre: server umount lustre-MDT0000 complete [10283.492535] LustreError: 272508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10283.513946] LustreError: 272508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10285.338279] LustreError: 272494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788682740 with bad export cookie 17903311271415031896 [10285.344056] LustreError: 250329:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9f2ece4e4e00 x1875569129254528/t0(0) o41->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 0 dl 1788682752 ref 1 fl Rpc:RESQU/200/0 rc -5/-107 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [10285.344537] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10285.352745] LustreError: 272494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10285.628579] Lustre: server umount lustre-MDT0001 complete [10299.500241] Lustre: server umount lustre-OST0000 complete [10314.085690] Lustre: server umount lustre-OST0001 complete [10328.329194] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10338.754950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10339.120855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10339.140096] Lustre: Skipped 3 previous similar messages [10343.569361] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10352.042473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10356.145121] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10358.592903] Lustre: 279302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10364.331607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10369.797167] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10377.957224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10384.220872] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10384.380522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10384.406108] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10391.652428] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10395.619319] Lustre: 281180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10400.985870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10401.028669] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10414.549446] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 04:21:08 (1788682868) [10429.408439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10429.417194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10429.432854] Lustre: Skipped 3 previous similar messages [10429.439947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10435.340885] Lustre: server umount lustre-MDT0000 complete [10435.564278] LustreError: 279254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10435.587978] LustreError: 279254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10438.564045] LustreError: 279682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788682893 with bad export cookie 17903311271415038287 [10438.572529] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10438.574514] LustreError: 279682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10445.033871] Lustre: server umount lustre-MDT0001 complete [10454.776467] Lustre: server umount lustre-OST0000 complete [10458.277589] Lustre: server umount lustre-OST0001 complete [10464.221722] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [10473.968454] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10523.433271] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10532.821161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10533.016714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10533.047349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10533.109814] Lustre: lustre-MDT0000: new disk, initializing [10533.190879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10537.922443] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10548.395608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10548.532326] Lustre: 285707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10548.840037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10554.030685] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10565.221254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10565.501500] Lustre: 287340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10567.342178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10572.553308] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10583.364976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10583.631171] Lustre: 288211:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10584.810555] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10584.818876] Lustre: Skipped 1 previous similar message [10584.883214] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10590.585522] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10599.267970] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10602.291867] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10602.296857] Lustre: Skipped 1 previous similar message [10617.791735] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 04:24:31 (1788683071) [10620.389669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10620.402208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10620.413678] Lustre: Skipped 5 previous similar messages [10620.417674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10620.424025] Lustre: Skipped 8 previous similar messages [10626.178840] Lustre: server umount lustre-MDT0000 complete [10629.370435] LustreError: 287350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788683084 with bad export cookie 17903311271415046253 [10629.375165] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10629.381850] LustreError: 287350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10629.583327] Lustre: server umount lustre-MDT0001 complete [10641.789361] Lustre: server umount lustre-OST0000 complete [10655.721701] Lustre: server umount lustre-OST0001 complete [10663.557674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10663.961633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10663.965836] Lustre: Skipped 7 previous similar messages [10667.865609] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10677.187030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.550305] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10690.105468] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10695.824638] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10701.855607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10703.282412] LustreError: 292283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10703.329806] LustreError: 292283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10707.343967] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10714.092935] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10728.026566] loop1: detected capacity change from 0 to 204800000 [10728.203772] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10728.211447] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10728.235719] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10732.107882] loop1: detected capacity change from 0 to 204800000 [10736.618769] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.665451] LustreError: 290445:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs31020: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10736.672107] LustreError: 290445:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs31020 [10736.678105] LustreError: 290445:0:(mgs_handler.c:517:mgs_target_reg()) fs31020-MDT0001: Error checking target index: rc = -13 [10736.688325] LustreError: fs31020-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10736.694850] LustreError: 294172:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [10736.701766] LustreError: 294172:0:(tgt_mount.c:2132:server_put_super()) no obd fs31020-MDT0001 [10736.704477] LustreError: 294172:0:(tgt_mount.c:116:server_deregister_mount()) fs31020-MDT0001 not registered [10736.729888] Lustre: server umount fs31020-MDT0001 complete [10736.731883] LustreError: 294172:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10744.241904] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10744.305783] Lustre: Setting parameter fs31020-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs31020-MDT0001 [10744.332326] Lustre: srv-fs31020-MDT0001: No data found on store. Initialize space. [10744.334779] Lustre: Skipped 4 previous similar messages [10744.383828] Lustre: fs31020-MDT0001: new disk, initializing [10744.385945] Lustre: Skipped 3 previous similar messages [10744.479478] Lustre: srv-fs31020-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10744.485674] Lustre: 294638:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs31020-MDT0001-mdtlov: can't create new llog: rc = -115 [10744.501205] LustreError: 294638:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs31020-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10748.413875] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10754.813749] Lustre: server umount fs31020-MDT0001 complete [10769.244219] loop1: detected capacity change from 0 to 204800000 [10774.659254] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10775.117100] Lustre: 296206:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs31020-MDT0001-mdtlov: can't create new llog: rc = -115 [10775.134576] LustreError: 296206:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs31020-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10779.785753] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10784.626909] Lustre: server umount fs31020-MDT0001 complete [10786.280878] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10795.488209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10795.498649] Lustre: Skipped 3 previous similar messages [10795.503802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10795.507224] Lustre: Skipped 7 previous similar messages [10806.239144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10806.381766] Lustre: server umount lustre-MDT0000 complete [10816.468497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10816.640776] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10821.376173] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10822.135348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10829.637520] loop1: detected capacity change from 0 to 204800000 [10835.224392] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10835.550768] Lustre: 298656:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs31020-MDT0001-mdtlov: can't create new llog: rc = -115 [10835.556390] LustreError: 298656:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs31020-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10840.124231] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10844.904709] Lustre: server umount fs31020-MDT0001 complete [10867.168613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10867.176848] LustreError: Skipped 2 previous similar messages [10867.185827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10867.191677] Lustre: Skipped 8 previous similar messages [10873.654465] Lustre: server umount lustre-OST0000 complete [10883.365372] Lustre: server umount lustre-MDT0000 complete [10886.883513] LustreError: 297199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788683342 with bad export cookie 17903311271415048934 [10886.888583] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10886.895774] LustreError: 297199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10887.330232] Lustre: server umount lustre-MDT0001 complete [10900.009889] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [10903.111268] Key type lgssc unregistered [10903.503530] LNet: 300998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10903.520778] LNetError: 300998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10903.548616] LNet: Removed LNI 192.168.204.142@tcp [10904.676182] Key type .llcrypt unregistered [10904.678967] Key type ._llcrypt unregistered [10928.439078] Key type ._llcrypt registered [10928.440430] Key type .llcrypt registered [10928.619280] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10930.237844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10930.281345] alg: No test for adler32 (adler32-zlib) [10931.303736] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [10931.656103] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [10933.489897] Key type lgssc registered [10934.901438] Lustre: Echo OBD driver; http://www.lustre.org/ [10943.107833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10943.144442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10944.698875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10948.759864] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10955.242972] LustreError: 302270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10955.267481] LustreError: 302270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10957.939632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10958.362759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10963.739756] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10971.372697] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10977.243897] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10984.430401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10985.031467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10991.149287] LustreError: 304096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10994.293913] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10996.207486] LustreError: 304098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11001.338783] LustreError: 304096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11004.168782] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11006.931954] LustreError: 304323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11006.951131] LustreError: 304323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11017.254139] LustreError: 304323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11017.284800] LustreError: 304323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11023.023693] loop1: detected capacity change from 0 to 204800000 [11023.250761] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11023.279623] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11023.297996] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11028.207676] loop1: detected capacity change from 0 to 204800000 [11032.811136] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11032.932606] Lustre: fs30541-OST0000: new disk, initializing [11032.937209] Lustre: srv-fs30541-OST0000: No data found on store. Initialize space. [11032.952269] Lustre: 306001:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11033.036357] Lustre: fs30541-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11034.085531] LustreError: 304096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11034.120123] LustreError: 304096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11039.080673] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11048.295767] Lustre: server umount fs30541-OST0000 complete [11060.665955] loop1: detected capacity change from 0 to 204800000 [11066.726677] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11066.964157] Lustre: fs30541-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11068.453107] LustreError: 304096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11068.485409] LustreError: 304096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11069.921833] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11073.390881] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11082.225232] loop2: detected capacity change from 0 to 204800000 [11082.463584] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11082.481187] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11082.495925] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11085.290994] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11087.927324] loop2: detected capacity change from 0 to 204800000 [11094.088983] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11094.228420] LustreError: fs30541-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11094.246141] LustreError: fs30541-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11094.261821] LustreError: 309042:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11094.274977] LustreError: 309042:0:(tgt_mount.c:2132:server_put_super()) no obd fs30541-OST0001 [11094.282463] LustreError: 309042:0:(tgt_mount.c:116:server_deregister_mount()) fs30541-OST0001 not registered [11094.320979] Lustre: server umount fs30541-OST0001 complete [11094.327719] LustreError: 309042:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11105.971258] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11106.176429] Lustre: fs30541-OST0001: new disk, initializing [11106.182204] Lustre: srv-fs30541-OST0001: No data found on store. Initialize space. [11106.189898] Lustre: 309592:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11106.286814] Lustre: fs30541-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11113.369887] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11122.358876] Lustre: server umount fs30541-OST0000 complete [11129.767459] Lustre: server umount fs30541-OST0001 complete [11135.026778] LustreError: 304323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11135.050572] LustreError: 304323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11141.238785] loop1: detected capacity change from 0 to 204800000 [11146.761626] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11146.969104] Lustre: fs30541-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11152.588834] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11158.867859] loop2: detected capacity change from 0 to 204800000 [11164.113944] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11170.390938] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11178.251223] Lustre: server umount fs30541-OST0000 complete [11186.616901] Lustre: server umount fs30541-OST0001 complete [11197.929758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11197.929948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11197.972450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11197.981525] Lustre: Skipped 1 previous similar message [11202.768324] Lustre: server umount lustre-OST0000 complete [11208.164651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11208.179855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11208.185378] Lustre: Skipped 2 previous similar messages [11212.377853] Lustre: server umount lustre-MDT0000 complete [11216.032380] LustreError: 310340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788683671 with bad export cookie 4240139922410709210 [11216.038100] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11216.042886] LustreError: 310340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11216.440295] Lustre: server umount lustre-MDT0001 complete [11225.718098] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [11228.104327] Key type lgssc unregistered [11228.323878] LNet: 315514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11228.329579] LNetError: 315514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11228.346813] LNet: Removed LNI 192.168.204.142@tcp [11228.997326] Key type .llcrypt unregistered [11228.999886] Key type ._llcrypt unregistered [11259.018323] Key type ._llcrypt registered [11259.021895] Key type .llcrypt registered [11259.156294] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [11281.618748] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11282.580451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11282.660068] alg: No test for adler32 (adler32-zlib) [11283.600218] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [11283.770643] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [11285.399526] Key type lgssc registered [11286.773417] Lustre: Echo OBD driver; http://www.lustre.org/ [11321.559568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11321.609937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11322.897637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11322.945897] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11323.035978] Lustre: lustre-MDT0000: new disk, initializing [11323.148902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11323.165665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11327.845401] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11339.018553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11339.153365] Lustre: 319372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11339.222983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11339.229802] Lustre: Skipped 1 previous similar message [11339.344738] Lustre: lustre-MDT0001: new disk, initializing [11339.471298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11339.514226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11339.539346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11344.434393] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11354.450488] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11360.004781] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11366.401824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11366.714822] Lustre: lustre-OST0000: new disk, initializing [11366.720431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11366.729569] Lustre: 321340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11366.838380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11372.598821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11372.621592] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11372.731183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11373.078995] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11381.970949] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11383.897218] Lustre: Mounted lustre-client - version 2.17.58_2_g2c24a36 [11384.759617] Lustre: Unmounted lustre-client [11393.520935] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11393.521258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11393.535528] LustreError: Skipped 1 previous similar message [11393.536434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11393.562033] Lustre: Skipped 1 previous similar message [11399.145048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11399.150234] Lustre: Skipped 4 previous similar messages [11399.642416] Lustre: server umount lustre-OST0000 complete [11404.256902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11404.265664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11404.268895] Lustre: Skipped 1 previous similar message [11404.284229] Lustre: Skipped 1 previous similar message [11409.379382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11409.389330] Lustre: Skipped 1 previous similar message [11410.003298] Lustre: server umount lustre-MDT0000 complete [11413.817179] LustreError: 319361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788683869 with bad export cookie 14326328340263462524 [11413.818376] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11413.825985] LustreError: 319361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11414.260344] Lustre: server umount lustre-MDT0001 complete [11425.600252] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [11429.079353] Key type lgssc unregistered [11439.455813] LNet: 1 local NIs in recovery (showing 1): 192.168.204.142@tcp [11439.713503] LNet: 323503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11439.724639] LNetError: 323503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11515.558357] LNet: Removed LNI 192.168.204.142@tcp [11516.574876] Key type .llcrypt unregistered [11516.576924] Key type ._llcrypt unregistered [11534.100990] Key type ._llcrypt registered [11534.105221] Key type .llcrypt registered [11534.204313] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [11549.357398] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11550.478202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11550.499649] alg: No test for adler32 (adler32-zlib) [11551.632871] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [11551.873243] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [11553.576428] Key type lgssc registered [11554.831591] Lustre: Echo OBD driver; http://www.lustre.org/ [11586.486664] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:40:40 (1788684040) [11593.361950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11593.383735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11594.714123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11594.780766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11594.893042] Lustre: lustre-MDT0000: new disk, initializing [11594.983303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11595.003670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11599.992620] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11611.840820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11611.948957] Lustre: 326719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11611.994105] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11611.997910] Lustre: Skipped 1 previous similar message [11612.098633] Lustre: lustre-MDT0001: new disk, initializing [11612.195312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11612.226055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11612.234049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11617.445339] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11628.500784] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11635.229780] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11642.513219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11642.797968] Lustre: lustre-OST0000: new disk, initializing [11642.802225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11642.811893] Lustre: 328690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11642.976139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11649.873204] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11653.219166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11653.245587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11653.449908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11660.323062] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11679.203301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11679.209718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11679.214790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11679.230093] Lustre: Skipped 1 previous similar message [11679.241932] Lustre: Skipped 1 previous similar message [11683.669727] Lustre: server umount lustre-OST0000 complete [11688.931154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11688.947560] LustreError: Skipped 1 previous similar message [11688.956414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11688.972769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11693.912948] Lustre: server umount lustre-MDT0000 complete [11698.029415] LustreError: 326713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684153 with bad export cookie 15793874786051553970 [11698.031024] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11698.049873] LustreError: 326713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11698.566164] Lustre: server umount lustre-MDT0001 complete [11709.803669] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [11713.096364] Key type lgssc unregistered [11713.450596] LNet: 330598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11713.463050] LNetError: 330598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11713.480855] LNet: Removed LNI 192.168.204.142@tcp [11714.436100] Key type .llcrypt unregistered [11714.439249] Key type ._llcrypt unregistered [11736.498924] Key type ._llcrypt registered [11736.501804] Key type .llcrypt registered [11736.605169] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11737.932111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11738.019418] alg: No test for adler32 (adler32-zlib) [11739.069989] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [11739.323827] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [11741.007392] Key type lgssc registered [11742.042912] Lustre: Echo OBD driver; http://www.lustre.org/ [11749.783412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11749.795642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11751.316159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11756.047903] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11760.622062] LustreError: 331868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.646475] LustreError: 331868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11764.877301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11765.409626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11769.550455] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11776.132992] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11781.461304] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11788.915873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11789.279746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11796.820234] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11805.869223] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11815.843766] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11817.520505] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11822.054854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11822.060140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11822.068724] Lustre: Skipped 1 previous similar message [11822.086481] Lustre: Skipped 2 previous similar messages [11823.188852] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [11827.171384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11832.294156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11832.315914] Lustre: Skipped 5 previous similar messages [11836.386781] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11836.604666] Lustre: server umount lustre-MDT0000 complete [11837.415413] LustreError: 334414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11837.445882] LustreError: 334414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11838.511316] LustreError: 334413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11840.648580] LustreError: 331850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684296 with bad export cookie 12760727098579871211 [11840.661654] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11842.536129] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11842.551585] LustreError: 331863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11842.553091] Lustre: Skipped 1 previous similar message [11842.560177] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11842.600543] Lustre: Skipped 1 previous similar message [11847.060442] Lustre: server umount lustre-MDT0001 complete [11851.334890] Lustre: server umount lustre-OST0000 complete [11861.005916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11867.226355] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11870.946856] Lustre: 336049:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.142@tcp [11873.722346] Lustre: server umount MGS complete [11883.187201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11883.732239] LustreError: 336623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11883.763548] LustreError: 336623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11883.841348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11888.623943] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11894.245022] LustreError: 336624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11894.273388] LustreError: 336624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11897.160029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11897.513510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11902.025259] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11910.893995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11911.367067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11918.987577] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11925.999704] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11927.554250] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11932.528280] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11934.000605] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11939.300717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11939.302097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11939.310885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11939.310895] Lustre: Skipped 2 previous similar messages [11939.317265] Lustre: Skipped 1 previous similar message [11940.034742] Lustre: server umount lustre-OST0000 complete [11944.417708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11944.436067] Lustre: Skipped 1 previous similar message [11950.541559] Lustre: server umount lustre-MDT0000 complete [11954.421217] LustreError: 336603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684409 with bad export cookie 12760727098579872058 [11954.427485] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11954.430496] LustreError: 336603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11954.832068] Lustre: server umount lustre-MDT0001 complete [11965.768795] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [11968.613781] Key type lgssc unregistered [11968.999138] LNet: 340058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11969.006996] LNetError: 340058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11969.023710] LNet: Removed LNI 192.168.204.142@tcp [11969.975888] Key type .llcrypt unregistered [11969.978914] Key type ._llcrypt unregistered [11988.028506] Key type ._llcrypt registered [11988.032312] Key type .llcrypt registered [11988.169625] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [12004.407854] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12005.283996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12005.418240] alg: No test for adler32 (adler32-zlib) [12006.444813] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [12006.739515] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12008.415319] Key type lgssc registered [12009.431183] Lustre: Echo OBD driver; http://www.lustre.org/ [12048.459219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12048.504930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12049.803948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12049.844426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12049.939561] Lustre: lustre-MDT0000: new disk, initializing [12050.030146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12050.049776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12055.427576] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12066.387888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12066.573328] Lustre: 343213:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12066.676510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12066.683263] Lustre: Skipped 1 previous similar message [12066.746528] Lustre: lustre-MDT0001: new disk, initializing [12066.836773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12066.884191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12066.898489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12071.193472] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12081.681771] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12087.985861] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12094.334374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12094.553651] Lustre: lustre-OST0000: new disk, initializing [12094.556865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12094.563192] Lustre: 345181:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12094.635784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12100.617922] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12104.294321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12104.310374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12104.421388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12111.327432] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12117.453538] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12117.863634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12123.716880] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12123.993320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12130.276282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12130.280483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12130.282549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12130.289461] LustreError: Skipped 1 previous similar message [12130.339072] Lustre: Skipped 1 previous similar message [12132.120320] Lustre: server umount lustre-OST0000 complete [12138.464683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12138.473168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12138.499756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12138.506345] Lustre: Skipped 1 previous similar message [12140.512466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12140.515857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12140.541220] Lustre: Skipped 1 previous similar message [12142.398093] Lustre: server umount lustre-MDT0000 complete [12146.056845] LustreError: 343206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684601 with bad export cookie 12163912596428605853 [12146.067066] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12146.071112] LustreError: 343206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12146.310926] Lustre: server umount lustre-MDT0001 complete [12155.863480] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:50:09 (1788684609) [12172.007621] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [12182.502928] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12217.072357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12217.354294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12217.407834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12217.534841] Lustre: lustre-MDT0000: new disk, initializing [12217.649052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12217.674390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12221.807700] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12233.412541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12233.597932] Lustre: 349944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12233.712841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12233.722894] Lustre: Skipped 1 previous similar message [12233.866785] Lustre: lustre-MDT0001: new disk, initializing [12234.026087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12234.084350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12234.097453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12239.530433] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12252.101366] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12259.186389] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12266.296251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12266.607519] Lustre: lustre-OST0000: new disk, initializing [12266.611501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12266.618035] Lustre: 351914:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12266.691057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12273.887946] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12275.308582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12275.332746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12275.466290] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12285.026146] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12321.764701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12321.769177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12321.796964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12326.635468] Lustre: server umount lustre-OST0000 complete [12331.493692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12331.510401] LustreError: Skipped 1 previous similar message [12331.517759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12331.543319] Lustre: Skipped 1 previous similar message [12331.553855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12331.558879] Lustre: Skipped 1 previous similar message [12337.120289] LustreError: 349957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12337.135630] LustreError: 349957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12337.228183] Lustre: server umount lustre-MDT0000 complete [12341.103238] LustreError: 351924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684796 with bad export cookie 12163912596428606735 [12341.112021] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12341.115989] LustreError: 351924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12341.401293] Lustre: server umount lustre-MDT0001 complete [12353.316603] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12356.754863] Key type lgssc unregistered [12357.068062] LNet: 354087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12357.077502] LNetError: 354087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12357.104267] LNet: Removed LNI 192.168.204.142@tcp [12358.104837] Key type .llcrypt unregistered [12358.109435] Key type ._llcrypt unregistered [12462.701608] Key type ._llcrypt registered [12462.704051] Key type .llcrypt registered [12462.840239] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12463.251674] Key type .llcrypt unregistered [12463.256882] Key type ._llcrypt unregistered [12547.634706] Key type ._llcrypt registered [12547.637963] Key type .llcrypt registered [12547.743272] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12548.044371] Key type .llcrypt unregistered [12548.046123] Key type ._llcrypt unregistered [12580.844862] Key type ._llcrypt registered [12580.847216] Key type .llcrypt registered [12580.992957] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12581.333416] Key type .llcrypt unregistered [12581.338507] Key type ._llcrypt unregistered [12611.805558] Key type ._llcrypt registered [12611.808327] Key type .llcrypt registered [12611.954765] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12612.324636] Key type .llcrypt unregistered [12612.327487] Key type ._llcrypt unregistered [12692.384396] Key type ._llcrypt registered [12692.392052] Key type .llcrypt registered [12692.518420] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12692.991239] Key type .llcrypt unregistered [12692.994210] Key type ._llcrypt unregistered [12726.048295] Key type ._llcrypt registered [12726.049929] Key type .llcrypt registered [12726.161041] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12726.531662] Key type .llcrypt unregistered [12726.533633] Key type ._llcrypt unregistered [12756.436787] Key type ._llcrypt registered [12756.440922] Key type .llcrypt registered [12756.573289] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12756.853375] Key type .llcrypt unregistered [12756.855662] Key type ._llcrypt unregistered [12789.590241] Key type ._llcrypt registered [12789.594811] Key type .llcrypt registered [12789.733064] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12790.011499] Key type .llcrypt unregistered [12790.014358] Key type ._llcrypt unregistered [12828.556138] Key type ._llcrypt registered [12828.559662] Key type .llcrypt registered [12828.737184] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12829.180549] Key type .llcrypt unregistered [12829.183544] Key type ._llcrypt unregistered [12868.711809] Key type ._llcrypt registered [12868.715042] Key type .llcrypt registered [12868.840627] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12869.239806] Key type .llcrypt unregistered [12869.245653] Key type ._llcrypt unregistered [12943.586278] Key type ._llcrypt registered [12943.588535] Key type .llcrypt registered [12943.728504] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12944.162487] Key type .llcrypt unregistered [12944.168669] Key type ._llcrypt unregistered [12978.001175] Key type ._llcrypt registered [12978.006760] Key type .llcrypt registered [12978.175596] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12978.504392] Key type .llcrypt unregistered [12978.506266] Key type ._llcrypt unregistered [13017.086937] Key type ._llcrypt registered [13017.088830] Key type .llcrypt registered [13017.234867] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13017.621391] Key type .llcrypt unregistered [13017.625060] Key type ._llcrypt unregistered [13079.105916] Key type ._llcrypt registered [13079.108305] Key type .llcrypt registered [13079.325709] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [13099.935323] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [13101.688419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13102.026766] alg: No test for adler32 (adler32-zlib) [13103.503610] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [13103.936513] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [13105.760322] Key type lgssc registered [13107.318756] Lustre: Echo OBD driver; http://www.lustre.org/ [13147.385642] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13147.411447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13148.795761] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13148.823466] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13148.909339] Lustre: tst250fs-MDT0000: new disk, initializing [13149.010329] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13149.031388] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13154.698748] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13167.448146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13167.712821] Lustre: 374398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13167.790981] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13167.806497] Lustre: Skipped 1 previous similar message [13167.963237] Lustre: tst250fs-MDT0001: new disk, initializing [13168.056781] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13168.107531] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13168.124861] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13174.003403] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13179.668118] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13188.910287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13189.237307] Lustre: tst250fs-OST0000: new disk, initializing [13189.244223] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13189.251521] Lustre: 376031:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13189.375507] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13191.096552] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13191.123962] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13191.337032] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13198.996427] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13212.235367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13212.452825] Lustre: tst250fs-OST0001: new disk, initializing [13212.456066] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13212.461415] Lustre: 376902:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13212.574306] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13220.898416] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13221.538941] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13221.562045] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13221.838808] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13227.768352] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13229.841604] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13229.853918] Lustre: Skipped 1 previous similar message [13252.587153] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13252.605972] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13252.616226] Lustre: Skipped 3 previous similar messages [13252.654830] Lustre: Skipped 3 previous similar messages [13256.850083] Lustre: server umount tst250fs-MDT0000 complete [13260.998102] LustreError: 374389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788685716 with bad export cookie 11898563924991213338 [13261.007663] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13261.016873] LustreError: 374389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13261.347055] Lustre: server umount tst250fs-MDT0001 complete [13276.343814] Lustre: server umount tst250fs-OST0000 complete [13279.201321] Lustre: 372621:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788685718/real 1788685718] req@ffff9f2ecaca1c00 x1875572748789504/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788685734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13279.216977] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13280.678766] Lustre: server umount tst250fs-OST0001 complete [13302.763424] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [13314.146128] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [13354.083566] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 05:10:07 (1788685807) [13356.034049] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13377.447462] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [13389.206120] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [13421.810775] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13103 sec ========= 05:11:15 (1788685875) [13423.692418] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:11:17 (1788685877) === [13427.757886] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:11:21 (1788685881) === [13472.071781] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13475.210170] Key type lgssc unregistered [13475.580651] LNet: 386054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13475.585936] LNetError: 386054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13475.599925] LNet: Removed LNI 192.168.204.142@tcp [13476.486159] Key type .llcrypt unregistered [13476.489587] Key type ._llcrypt unregistered