[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478602765 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.003266] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.009026] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011149] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013046] SELinux: Initializing. [ 0.014083] *** VALIDATE selinux *** [ 0.023481] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028242] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031130] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032120] *** VALIDATE tmpfs *** [ 0.034287] *** VALIDATE proc *** [ 0.035259] *** VALIDATE cgroup *** [ 0.037008] *** VALIDATE cgroup2 *** [ 0.038290] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041033] Spectre V2 : User space: Vulnerable [ 0.043011] Speculative Store Bypass: Vulnerable [ 0.046308] debug: unmapping init [mem 0xffffffff9ee59000-0xffffffff9ee60fff] [ 0.048187] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049675] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050024] ... version: 2 [ 0.051014] ... bit width: 48 [ 0.052018] ... generic registers: 4 [ 0.053017] ... value mask: 0000ffffffffffff [ 0.054018] ... max period: 00007fffffffffff [ 0.055017] ... fixed-purpose events: 3 [ 0.056015] ... event mask: 000000070000000f [ 0.057269] rcu: Hierarchical SRCU implementation. [ 0.059562] smp: Bringing up secondary CPUs ... [ 0.060750] x86: Booting SMP configuration: [ 0.061029] .... node #0, CPUs: #1 #2 #3 [ 0.064460] smp: Brought up 1 node, 4 CPUs [ 0.066011] smpboot: Max logical packages: 1 [ 0.067018] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.142502] node 0 deferred pages initialised in 71ms [ 0.147029] devtmpfs: initialized [ 0.148312] x86/mm: Memory block size: 128MB [ 0.149000] gcov: version magic: 0x41383552 [ 0.151220] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157353] pinctrl core: initialized pinctrl subsystem [ 0.158183] [ 0.158751] ************************************************************* [ 0.161013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164013] ** ** [ 0.166011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168009] ** ** [ 0.170010] ** This means that this kernel is built to expose internal ** [ 0.172015] ** IOMMU data structures, which may compromise security on ** [ 0.175012] ** your system. ** [ 0.177012] ** ** [ 0.179014] ** If you see this message and you are not debugging the ** [ 0.183013] ** kernel, report this immediately to your vendor! ** [ 0.185011] ** ** [ 0.187011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189013] ************************************************************* [ 0.191691] NET: Registered protocol family 16 [ 0.193422] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203115] cpuidle: using governor menu [ 0.205012] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.207542] PCI: Using configuration type 1 for base access [ 0.209137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.219161] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.222105] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225087] cryptd: max_cpu_qlen set to 1000 [ 0.227248] ACPI: Added _OSI(Module Device) [ 0.229014] ACPI: Added _OSI(Processor Device) [ 0.230012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237080] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242519] ACPI: Interpreter enabled [ 0.244063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.245010] ACPI: Using IOAPIC for interrupt routing [ 0.247129] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.250388] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.260423] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.265017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.268095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273243] acpiphp: Slot [2] registered [ 0.274093] acpiphp: Slot [5] registered [ 0.275077] acpiphp: Slot [6] registered [ 0.277130] acpiphp: Slot [7] registered [ 0.278104] acpiphp: Slot [8] registered [ 0.279115] acpiphp: Slot [9] registered [ 0.280092] acpiphp: Slot [10] registered [ 0.281093] acpiphp: Slot [3] registered [ 0.283125] acpiphp: Slot [4] registered [ 0.284128] acpiphp: Slot [11] registered [ 0.286099] acpiphp: Slot [12] registered [ 0.287104] acpiphp: Slot [13] registered [ 0.288138] acpiphp: Slot [14] registered [ 0.290111] acpiphp: Slot [15] registered [ 0.292136] acpiphp: Slot [16] registered [ 0.294131] acpiphp: Slot [17] registered [ 0.295074] acpiphp: Slot [18] registered [ 0.296110] acpiphp: Slot [19] registered [ 0.298117] acpiphp: Slot [20] registered [ 0.299100] acpiphp: Slot [21] registered [ 0.301193] acpiphp: Slot [22] registered [ 0.303126] acpiphp: Slot [23] registered [ 0.304133] acpiphp: Slot [24] registered [ 0.305035] acpiphp: Slot [25] registered [ 0.306104] acpiphp: Slot [26] registered [ 0.308120] acpiphp: Slot [27] registered [ 0.309093] acpiphp: Slot [28] registered [ 0.310080] acpiphp: Slot [29] registered [ 0.312112] acpiphp: Slot [30] registered [ 0.313117] acpiphp: Slot [31] registered [ 0.315065] PCI host bridge to bus 0000:00 [ 0.316020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.319027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.321035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.324031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.326024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.327020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329224] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.332689] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.334900] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.344587] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.349491] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.351020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.354025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.356021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.359605] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.362796] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.365060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.368796] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.373018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.383008] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.388014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.393682] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.400020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.408014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.428016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.438789] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.445018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.452018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.473023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.488130] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.496016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.504017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.522017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.533134] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.542022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.549024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.571019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.583994] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.590017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.600021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.628023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.640551] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.647016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.656021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.688023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.703154] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.706457] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.709400] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.712464] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.714230] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.720301] iommu: Default domain type: Passthrough [ 0.721415] SCSI subsystem initialized [ 0.723030] ACPI: bus type USB registered [ 0.725198] usbcore: registered new interface driver usbfs [ 0.727112] usbcore: registered new interface driver hub [ 0.729102] usbcore: registered new device driver usb [ 0.731240] pps_core: LinuxPPS API ver. 1 registered [ 0.732014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.735085] PTP clock support registered [ 0.737213] EDAC MC: Ver: 3.0.0 [ 0.739164] PCI: Using ACPI for IRQ routing [ 0.742098] NetLabel: Initializing [ 0.743000] NetLabel: domain hash size = 128 [ 0.743000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.745116] NetLabel: unlabeled traffic allowed by default [ 0.747161] vgaarb: loaded [ 0.749357] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.751017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.758717] clocksource: Switched to clocksource kvm-clock [ 0.872636] VFS: Disk quotas dquot_6.6.0 [ 0.874420] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.877122] *** VALIDATE ramfs *** [ 0.878600] *** VALIDATE hugetlbfs *** [ 0.880397] pnp: PnP ACPI init [ 0.883038] pnp: PnP ACPI: found 6 devices [ 0.905043] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.908353] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.910433] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.912298] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.914836] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.917675] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.920791] NET: Registered protocol family 2 [ 0.923311] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.928687] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.932586] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.937966] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.941761] TCP: Hash tables configured (established 65536 bind 65536) [ 0.945226] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.948295] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.950786] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.953674] NET: Registered protocol family 1 [ 0.956586] RPC: Registered named UNIX socket transport module. [ 0.959703] RPC: Registered udp transport module. [ 0.961663] RPC: Registered tcp transport module. [ 0.963697] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.966149] NET: Registered protocol family 44 [ 0.967756] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.969383] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.971551] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.974137] PCI: CLS 0 bytes, default 64 [ 0.976460] Unpacking initramfs... [ 2.405668] debug: unmapping init [mem 0xffff9b443cc54000-0xffff9b443ffbffff] [ 2.411933] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.414622] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.418305] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 2.908388] Initialise system trusted keyrings [ 2.910436] Key type blacklist registered [ 2.912579] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.924068] zbud: loaded [ 2.928524] *** VALIDATE nfs *** [ 2.929807] *** VALIDATE nfs4 *** [ 2.931539] pstore: using deflate compression [ 2.934913] Platform Keyring initialized [ 3.043882] NET: Registered protocol family 38 [ 3.045824] Key type asymmetric registered [ 3.047752] Asymmetric key parser 'x509' registered [ 3.049846] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.053505] io scheduler mq-deadline registered [ 3.055304] io scheduler kyber registered [ 3.057190] io scheduler bfq registered [ 3.059831] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.063510] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.066782] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.069978] ACPI: Power Button [PWRF] [ 3.075732] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.083225] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.098757] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.105974] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.123892] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.151380] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.181313] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.187398] Non-volatile memory driver v1.3 [ 3.189381] Linux agpgart interface v0.103 [ 3.227208] virtio_blk virtio1: [vda] 135296 512-byte logical blocks (69.3 MB/66.1 MiB) [ 3.230394] vda: detected capacity change from 0 to 69271552 [ 3.249307] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.251956] vdb: detected capacity change from 0 to 1073741824 [ 3.266172] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.268744] vdc: detected capacity change from 0 to 2621440000 [ 3.285127] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.287792] vdd: detected capacity change from 0 to 2621440000 [ 3.303195] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.305784] vde: detected capacity change from 0 to 4294967296 [ 3.320267] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.322904] vdf: detected capacity change from 0 to 4294967296 [ 3.331784] libphy: Fixed MDIO Bus: probed [ 3.346389] usbcore: registered new interface driver usbserial_generic [ 3.349351] usbserial: USB Serial support registered for generic [ 3.351832] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.358164] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.360309] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.362979] mousedev: PS/2 mouse device common for all mice [ 3.365530] rtc_cmos 00:05: RTC can wake from S4 [ 3.368366] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.370177] rtc_cmos 00:05: registered as rtc0 [ 3.373368] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.375062] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.377109] intel_pstate: CPU model not supported [ 3.382521] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.385806] hid: raw HID events driver (C) Jiri Kosina [ 3.388145] usbcore: registered new interface driver usbhid [ 3.389888] usbhid: USB HID core driver [ 3.391451] drop_monitor: Initializing network drop monitor service [ 3.394364] Initializing XFRM netlink socket [ 3.396987] NET: Registered protocol family 10 [ 3.400698] Segment Routing with IPv6 [ 3.402700] NET: Registered protocol family 17 [ 3.405074] mpls_gso: MPLS GSO support [ 3.411785] RAS: Correctable Errors collector initialized. [ 3.414266] AVX version of gcm_enc/dec engaged. [ 3.416129] AES CTR mode by8 optimization enabled [ 3.500412] sched_clock: Marking stable (3500384152, 0)->(4439042937, -938658785) [ 3.504140] registered taskstats version 1 [ 3.506519] Loading compiled-in X.509 certificates [ 3.508720] zswap: loaded using pool lzo/zbud [ 3.534081] Key type big_key registered [ 3.546560] Key type encrypted registered [ 3.547991] ima: No TPM chip found, activating TPM-bypass! [ 3.550429] ima: Allocated hash algorithm: sha1 [ 3.552202] ima: No architecture policies found [ 3.554159] evm: Initialising EVM extended attributes: [ 3.556532] evm: security.selinux [ 3.557669] evm: security.ima [ 3.558520] evm: security.capability [ 3.559782] evm: HMAC attrs: 0x1 [ 3.561932] rtc_cmos 00:05: setting system clock to 2026-05-28 12:49:34 UTC (1779972574) [ 3.568775] debug: unmapping init [mem 0xffffffff9fe03000-0xffffffff9fffffff] [ 3.572480] debug: unmapping init [mem 0xffffffff9eb82000-0xffffffff9ee58fff] [ 3.582102] Write protecting the kernel read-only data: 28672k [ 3.585632] debug: unmapping init [mem 0xffffffff9d203000-0xffffffff9d3fffff] [ 3.588642] debug: unmapping init [mem 0xffffffff9db14000-0xffffffff9dbfffff] [ 3.621909] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.630777] systemd[1]: Detected virtualization kvm. [ 3.633367] systemd[1]: Detected architecture x86-64. [ 3.641742] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.673329] systemd[1]: No hostname configured. [ 3.675334] systemd[1]: Set hostname to . [ 3.677754] random: systemd: uninitialized urandom read (16 bytes read) [ 3.680179] systemd[1]: Initializing machine ID from random generator. [ 3.726195] random: ln: uninitialized urandom read (6 bytes read) [ 3.804123] random: systemd: uninitialized urandom read (16 bytes read) [ 3.807737] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.813343] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.817806] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.430868] device-mapper: uevent: version 1.0.3 [ 4.433624] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.134498] random: fast init done [ 5.169078] virtio_net virtio0 ens2: renamed from eth0 [ 5.194925] scsi host0: ata_piix [ 5.283957] scsi host1: ata_piix [ 5.287573] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.290461] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.500352] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.063868] random: crng init done [ 10.065345] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.401185] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.575975] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.831774] SELinux: Disabled at runtime. [ 11.893513] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.905801] systemd[1]: Detected virtualization kvm. [ 11.908025] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.401958] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.407603] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.412387] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.416345] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.419595] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.429268] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.439345] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for det[ 12.542199] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ails. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.885396] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.193741] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.203083] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.334320] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.353821] EDAC sbridge: Ver: 1.1.2 [ 15.043780] Key type dns_resolver registered [ 15.354667] NFS: Registering the id_resolver key type [ 15.356968] Key type id_resolver registered [ 15.359017] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg454-server login: [ 50.264481] libcfs: loading out-of-tree module taints kernel. [ 50.322963] Key type ._llcrypt registered [ 50.327420] Key type .llcrypt registered [ 50.487908] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 69.241596] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 70.864421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.885953] alg: No test for adler32 (adler32-zlib) [ 72.268420] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 73.091374] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 74.831639] Key type lgssc registered [ 76.818038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.633959] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 100.031040] hrtimer: interrupt took 5771859 ns [ 142.118327] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 157.151775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 157.183052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.579629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 158.628401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 158.800467] Lustre: lustre-MDT0000: new disk, initializing [ 158.939431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.952842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 164.480234] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.269729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.383782] Lustre: 6520:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.406971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.411316] Lustre: Skipped 1 previous similar message [ 179.516819] Lustre: lustre-MDT0001: new disk, initializing [ 179.581281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.615507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.629872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.069150] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.419611] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.179648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.570559] Lustre: lustre-OST0000: new disk, initializing [ 201.577683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 201.669314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.238429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 203.252485] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 203.323899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 208.613882] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.292960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.529581] Lustre: lustre-OST0001: new disk, initializing [ 226.543777] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 226.635862] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 233.505533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 233.524731] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 233.640807] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 233.865418] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.703472] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.231972] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 270.985272] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing check_logdir /tmp/testlogs/ [ 276.051051] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing yml_node [ 281.068949] Lustre: DEBUG MARKER: Client: 2.17.50.143 [ 283.830798] Lustre: DEBUG MARKER: MDS: 2.17.50.143 [ 286.866583] Lustre: DEBUG MARKER: OSS: 2.17.50.143 [ 288.620396] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu May 28 08:54:16 EDT 2026 [ 310.804487] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 312.324467] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 328.675031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.698302] Lustre: Skipped 1 previous similar message [ 328.701629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.707756] Lustre: Skipped 2 previous similar messages [ 333.279888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 333.292674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.308859] Lustre: Skipped 1 previous similar message [ 333.314247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.048377] Lustre: server umount lustre-MDT0000 complete [ 338.662974] LustreError: 7461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779972909 with bad export cookie 963774511898023124 [ 338.668992] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.676348] LustreError: 7461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 338.916630] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.930517] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 338.935742] Lustre: Skipped 4 previous similar messages [ 344.032880] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.033155] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.056475] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 344.067583] Lustre: Skipped 1 previous similar message [ 345.001801] Lustre: server umount lustre-MDT0001 complete [ 349.588385] Lustre: server umount lustre-OST0000 complete [ 354.150838] Lustre: server umount lustre-OST0001 complete [ 362.467570] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 373.184930] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 412.360551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.651878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 412.683827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 412.808203] Lustre: lustre-MDT0000: new disk, initializing [ 412.927142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.950935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 417.927475] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 429.432877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.584446] Lustre: 15003:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 429.662232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 429.669931] Lustre: Skipped 1 previous similar message [ 429.797769] Lustre: lustre-MDT0001: new disk, initializing [ 429.902240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 429.937064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 429.947388] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 434.138255] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 442.594631] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 447.192552] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 455.769349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 456.421496] Lustre: lustre-OST0000: new disk, initializing [ 456.425938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 456.529229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.505887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 457.512930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 457.736304] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 463.882828] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 472.575646] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 477.337273] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 477.658757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 483.283989] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 483.597221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 488.443285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 488.448385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.460657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 488.489750] Lustre: Skipped 2 previous similar messages [ 488.510912] Lustre: Skipped 1 previous similar message [ 498.666086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 498.674162] Lustre: Skipped 3 previous similar messages [ 500.191140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 500.472721] Lustre: server umount lustre-OST0000 complete [ 506.850775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 506.861598] LustreError: Skipped 1 previous similar message [ 506.863689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.451416] Lustre: server umount lustre-MDT0000 complete [ 514.021744] LustreError: 15011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.037885] LustreError: 15011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 514.425635] LustreError: 16745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779973085 with bad export cookie 963774511898024657 [ 514.438080] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 514.444189] LustreError: 16745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 514.784103] Lustre: server umount lustre-MDT0001 complete [ 520.059547] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:58:08 (1779973088) [ 539.424286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 539.910852] LustreError: 19476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.969035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 544.080967] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 545.254740] LustreError: 19476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.352951] LustreError: 19476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.569655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.401504] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 562.802859] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 567.057564] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 569.283224] Lustre: Failing over lustre-MDT0000 [ 569.313237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.325502] Lustre: Skipped 1 previous similar message [ 569.329213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 569.337655] Lustre: Skipped 4 previous similar messages [ 569.814846] Lustre: server umount lustre-MDT0000 complete [ 574.433619] LustreError: 19471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.458196] LustreError: 19471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 590.570629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.773290] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 590.795822] Lustre: 3668:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779973145/real 1779973145] req@ffff9b4484b85500 x1866436595326976/t0(0) o400->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779973161 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 596.430433] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 596.452646] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 596.455123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 596.480828] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 605.987939] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 610.784864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.803300] Lustre: Skipped 1 previous similar message [ 610.811315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 610.830052] Lustre: Skipped 1 previous similar message [ 616.583620] Lustre: server umount lustre-MDT0000 complete [ 620.572162] LustreError: 21797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779973191 with bad export cookie 963774511898026071 [ 620.580495] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 620.588264] LustreError: 21797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 620.944499] Lustre: server umount lustre-MDT0001 complete [ 630.356124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.767530] LustreError: 22473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.794865] LustreError: 22473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 630.880329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 630.885089] Lustre: Skipped 2 previous similar messages [ 635.776571] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 644.832706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.278193] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 657.000313] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 661.166386] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 663.557058] Lustre: Failing over lustre-MDT0000 [ 663.581766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 663.593555] Lustre: Skipped 1 previous similar message [ 664.045401] Lustre: server umount lustre-MDT0000 complete [ 668.642168] LustreError: 23183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 668.665356] LustreError: 23183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 684.724578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 684.945662] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 685.379381] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 690.290606] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 690.470274] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 690.481805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 690.496479] Lustre: Skipped 1 previous similar message [ 690.513503] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 699.775598] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 706.019135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 706.024739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.053750] Lustre: Skipped 5 previous similar messages [ 710.665896] Lustre: server umount lustre-MDT0000 complete [ 714.077747] LustreError: 22455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779973285 with bad export cookie 963774511898027009 [ 714.088287] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 714.455723] Lustre: server umount lustre-MDT0001 complete [ 724.094950] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:01:32 (1779973292) [ 742.877381] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 753.493741] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 802.550860] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 816.077539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.338188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 816.374528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 816.494215] Lustre: lustre-MDT0000: new disk, initializing [ 816.631613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 816.652254] Lustre: Skipped 2 previous similar messages [ 816.669248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 822.298922] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 834.499859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 834.636644] Lustre: 28476:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 834.930550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 839.836432] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 847.534333] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 851.560726] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 859.612285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 861.238307] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 861.301220] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 861.449973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 861.490385] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 867.802647] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 877.069372] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 884.485152] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 912.881456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 912.885914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 912.894402] LustreError: Skipped 1 previous similar message [ 912.896503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 912.917668] Lustre: Skipped 4 previous similar messages [ 914.399937] Lustre: server umount lustre-OST0000 complete [ 922.082374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 924.745781] Lustre: server umount lustre-MDT0000 complete [ 928.225266] LustreError: 28487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 928.267453] LustreError: 28487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 929.398992] LustreError: 28469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779973500 with bad export cookie 963774511898027415 [ 929.411817] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 929.420588] LustreError: 28469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 930.058910] Lustre: server umount lustre-MDT0001 complete [ 943.232969] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 946.822356] Key type lgssc unregistered [ 947.191820] LNet: 32067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.205588] LNetError: 32067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.226383] LNet: Removed LNI 192.168.204.154@tcp [ 948.530175] Key type .llcrypt unregistered [ 948.532539] Key type ._llcrypt unregistered [ 974.541649] Key type ._llcrypt registered [ 974.545751] Key type .llcrypt registered [ 974.659157] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 976.942952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 977.014814] alg: No test for adler32 (adler32-zlib) [ 978.364219] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 978.706919] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 980.432462] Key type lgssc registered [ 982.316494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 993.312129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 993.343745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 995.147027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1001.771789] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1005.539759] LustreError: 33464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1005.560990] LustreError: 33464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1010.659178] LustreError: 33464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.750792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.305979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1019.601291] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1027.094454] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1031.858249] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1040.407344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1040.866605] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1041.077450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1046.558398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1048.357950] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1053.827311] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1058.805867] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1070.687376] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:07:19 (1779973639) [ 1072.645924] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1074.801652] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1082.023842] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:07:29 (1779973649) [ 1091.853932] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:07:40 (1779973660) [ 1092.996452] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1101.818228] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1101.829488] Lustre: Skipped 2 previous similar messages [ 1103.937890] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:07:51 (1779973671) [ 1105.876903] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1105.886393] Lustre: Skipped 2 previous similar messages [ 1125.117709] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1125.121754] Lustre: Skipped 7 previous similar messages [ 1128.636969] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:08:16 (1779973696) [ 1131.832821] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1135.514773] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1145.348429] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:08:32 (1779973712) [ 1147.190694] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1147.194588] Lustre: Skipped 3 previous similar messages [ 1163.566927] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:08:51 (1779973731) [ 1179.627085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1179.632909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.652143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1181.664737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1181.673409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.685202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1181.714285] Lustre: Skipped 1 previous similar message [ 1185.647787] Lustre: server umount lustre-OST0000 complete [ 1191.904369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.918663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1191.922551] Lustre: Skipped 1 previous similar message [ 1191.947887] Lustre: Skipped 1 previous similar message [ 1196.621322] Lustre: server umount lustre-MDT0000 complete [ 1197.031490] LustreError: 33464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.044119] LustreError: 33464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1201.128503] LustreError: 33464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.152372] LustreError: 33464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1201.255252] LustreError: 33446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779973772 with bad export cookie 8810906787161282955 [ 1201.260032] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1201.276354] LustreError: 33446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1201.749959] Lustre: server umount lustre-MDT0001 complete [ 1216.397935] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1222.114102] Key type lgssc unregistered [ 1222.562864] LNet: 40973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1222.581584] LNetError: 40973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1222.597605] LNet: Removed LNI 192.168.204.154@tcp [ 1223.640390] Key type .llcrypt unregistered [ 1223.644958] Key type ._llcrypt unregistered [ 1250.746514] Key type ._llcrypt registered [ 1250.751323] Key type .llcrypt registered [ 1250.938958] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 1269.597901] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1271.397296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1271.656370] alg: No test for adler32 (adler32-zlib) [ 1272.719161] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 1272.917745] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1274.575254] Key type lgssc registered [ 1276.553951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1322.196325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1322.285799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.657530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1323.708228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1323.810372] Lustre: lustre-MDT0000: new disk, initializing [ 1323.943406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1323.959919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1328.997455] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1342.692915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.792869] Lustre: 44099:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1342.897425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1342.906779] Lustre: Skipped 1 previous similar message [ 1343.063885] Lustre: lustre-MDT0001: new disk, initializing [ 1343.283444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1343.443638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1343.460182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1348.850598] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1358.397373] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1363.453051] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1371.669922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1371.930675] Lustre: lustre-OST0000: new disk, initializing [ 1371.934594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1372.047258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.375965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1378.387552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1378.511634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1378.600924] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1387.628546] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1393.235439] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1393.504888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1398.478592] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1398.673452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1404.388026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1404.394947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1404.418232] LustreError: Skipped 1 previous similar message [ 1404.423357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1404.482789] Lustre: Skipped 1 previous similar message [ 1409.518439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1409.536049] Lustre: Skipped 1 previous similar message [ 1414.645297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1414.656427] Lustre: Skipped 1 previous similar message [ 1415.135434] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1415.537402] Lustre: server umount lustre-OST0000 complete [ 1420.259565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1420.289316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.307740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1424.869557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1424.883171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1426.614855] Lustre: server umount lustre-MDT0000 complete [ 1431.886402] LustreError: 44878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779974002 with bad export cookie 1215083937650954086 [ 1431.888817] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1431.904580] LustreError: 44878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1432.420984] Lustre: server umount lustre-MDT0001 complete [ 1443.467984] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:13:31 (1779974011) [ 1450.749161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.307136] LustreError: 48158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1451.324717] LustreError: 48158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1451.416591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.472489] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1456.613760] LustreError: 48158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.730183] LustreError: 48158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1466.849414] LustreError: 48158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1466.860480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.180658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1472.796382] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1479.737907] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1487.188052] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1496.912169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1497.279691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.968209] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1512.904381] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1527.984467] LustreError: 50372:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1528.789193] Lustre: Modifying parameter general.timeout=1 in log params [ 1530.515947] Lustre: Modifying parameter general.timeout=3 in log params [ 1530.520071] Lustre: Skipped 1 previous similar message [ 1533.198114] Lustre: Modifying parameter general.timeout=6 in log params [ 1533.204165] Lustre: Skipped 2 previous similar messages [ 1537.979943] Lustre: Modifying parameter general.timeout=11 in log params [ 1537.982579] Lustre: Skipped 4 previous similar messages [ 1546.596701] Lustre: Modifying parameter general.timeout=20 in log params [ 1546.599702] Lustre: Skipped 8 previous similar messages [ 1563.355309] Lustre: Modifying parameter general.timeout=38 in log params [ 1563.361189] Lustre: Skipped 17 previous similar messages [ 1596.030820] Lustre: Modifying parameter general.timeout=72 in log params [ 1596.035299] Lustre: Skipped 33 previous similar messages [ 1661.350128] Lustre: Disabling parameter general.timeout= in log params [ 1661.356517] Lustre: Skipped 57 previous similar messages [ 1663.855989] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:17:12 (1779974232) [ 1667.195964] Lustre: 57326:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1669.919313] Lustre: 57424:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1672.554584] Lustre: 57535:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1676.575373] Lustre: 57732:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1676.592162] Lustre: 57732:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1719.786081] Lustre: 59211:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1719.799366] Lustre: 59211:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1719.819140] LustreError: 59211:0:(mgs_handler.c:1166:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1725.632791] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:18:13 (1779974293) [ 1742.818262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.829977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1742.846503] Lustre: Skipped 2 previous similar messages [ 1742.851952] Lustre: Skipped 2 previous similar messages [ 1746.025217] Lustre: server umount lustre-MDT0000 complete [ 1747.937287] LustreError: 48152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1747.964899] LustreError: 48152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1751.167431] LustreError: 48136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779974322 with bad export cookie 1215083937650954968 [ 1751.170928] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1751.183812] LustreError: 48136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1751.684625] Lustre: server umount lustre-MDT0001 complete [ 1767.355899] Lustre: server umount lustre-OST0000 complete [ 1796.841599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.095634] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1797.116507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1797.124768] Lustre: Skipped 22 previous similar messages [ 1797.387675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.470428] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1810.458747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1810.592080] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1810.634528] Lustre: 61746:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1816.182432] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1821.952371] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1825.800918] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1832.543316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1832.761125] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1833.054812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.059611] Lustre: Skipped 1 previous similar message [ 1839.051304] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1845.080291] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1847.327307] Lustre: 63970:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1857.508159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1857.516674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1857.527742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1857.535533] Lustre: Skipped 1 previous similar message [ 1860.730525] Lustre: server umount lustre-OST0000 complete [ 1867.262414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1867.281332] LustreError: Skipped 1 previous similar message [ 1867.302646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.327855] Lustre: Skipped 1 previous similar message [ 1871.852683] Lustre: server umount lustre-MDT0000 complete [ 1873.376951] LustreError: 61755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1873.393848] LustreError: 61755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1876.441780] LustreError: 61739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779974447 with bad export cookie 1215083937650958209 [ 1876.451748] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1876.457219] LustreError: 61739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1877.377156] Lustre: server umount lustre-MDT0001 complete [ 1887.979759] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1892.362447] Key type lgssc unregistered [ 1892.790774] LNet: 65165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1892.811731] LNetError: 65165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1892.825779] LNet: Removed LNI 192.168.204.154@tcp [ 1893.895279] Key type .llcrypt unregistered [ 1893.896873] Key type ._llcrypt unregistered [ 1921.592967] Key type ._llcrypt registered [ 1921.596392] Key type .llcrypt registered [ 1921.760641] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1923.063247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1923.113978] alg: No test for adler32 (adler32-zlib) [ 1924.199530] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 1924.428219] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1926.103167] Key type lgssc registered [ 1927.222924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1934.574755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1934.616993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1936.158807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.657750] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1945.576193] LustreError: 66563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1945.599578] LustreError: 66563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1950.306613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.156941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1956.473360] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1963.211189] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1967.732162] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1976.421156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1977.143866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.413672] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1990.469088] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2003.330594] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2016.229927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.233887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2016.255291] Lustre: Skipped 1 previous similar message [ 2021.357592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.365302] Lustre: Skipped 4 previous similar messages [ 2021.729136] Lustre: server umount lustre-MDT0000 complete [ 2025.481464] LustreError: 66545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779974596 with bad export cookie 5627689377773489020 [ 2025.489547] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.496469] LustreError: 66545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2025.908862] Lustre: server umount lustre-MDT0001 complete [ 2040.576403] Lustre: server umount lustre-OST0000 complete [ 2072.228562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.483698] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2072.526654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2073.024556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.959322] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2093.458921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.694929] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2093.793475] Lustre: 71104:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2094.233907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2100.007976] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2107.877067] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2113.276752] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2124.092741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2124.463915] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2125.255613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2135.161466] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2144.020561] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2146.693811] Lustre: 73345:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2148.736089] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2159.074994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.083505] Lustre: Skipped 2 previous similar messages [ 2159.085636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2159.086493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2159.109176] Lustre: Skipped 1 previous similar message [ 2160.813329] Lustre: server umount lustre-OST0000 complete [ 2165.729188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2165.741320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.768655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.317366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.730805] Lustre: server umount lustre-MDT0000 complete [ 2174.432530] LustreError: 71116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2174.639047] LustreError: 71098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779974745 with bad export cookie 5627689377773489867 [ 2174.639743] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2174.654569] LustreError: 71098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2175.011678] Lustre: server umount lustre-MDT0001 complete [ 2188.192456] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2191.596994] Key type lgssc unregistered [ 2192.021788] LNet: 74537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2192.035609] LNetError: 74537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2192.058776] LNet: Removed LNI 192.168.204.154@tcp [ 2192.983816] Key type .llcrypt unregistered [ 2192.986092] Key type ._llcrypt unregistered [ 2217.930754] Key type ._llcrypt registered [ 2217.932957] Key type .llcrypt registered [ 2218.117869] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2219.466458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2219.491666] alg: No test for adler32 (adler32-zlib) [ 2220.691275] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 2220.895956] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2222.591222] Key type lgssc registered [ 2223.768795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2232.950399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2232.991713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.676178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2240.029548] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2244.073789] LustreError: 75934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.099402] LustreError: 75934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2249.186268] LustreError: 75934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.243053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.916623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2255.468960] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2261.807368] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2265.538348] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2273.885917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2274.285179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2280.764527] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2286.724811] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2293.774232] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2294.888813] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2295.994921] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2298.957941] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2298.969770] Lustre: Skipped 2 previous similar messages [ 2303.696784] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2303.709342] Lustre: Skipped 4 previous similar messages [ 2311.817604] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2311.825973] Lustre: Skipped 7 previous similar messages [ 2328.934821] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2328.942079] Lustre: Skipped 12 previous similar messages [ 2362.210679] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2362.215471] Lustre: Skipped 29 previous similar messages [ 2410.446044] Lustre: 83711:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2413.451540] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:29:41 (1779974981) [ 2415.387476] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2417.445785] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:29:45 (1779974985) [ 2422.765661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.774939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2422.782050] Lustre: Skipped 1 previous similar message [ 2422.782683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2422.787629] Lustre: Skipped 1 previous similar message [ 2427.876143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2427.886244] Lustre: Skipped 1 previous similar message [ 2428.180308] Lustre: server umount lustre-OST0000 complete [ 2432.996084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2433.004584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2433.024967] Lustre: Skipped 1 previous similar message [ 2433.044120] Lustre: Skipped 1 previous similar message [ 2438.117648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2438.125293] Lustre: Skipped 1 previous similar message [ 2438.447702] Lustre: server umount lustre-MDT0000 complete [ 2441.885020] LustreError: 78884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779975012 with bad export cookie 4291040273361817534 [ 2441.888421] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2441.896322] LustreError: 78884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2442.278774] Lustre: server umount lustre-MDT0001 complete [ 2453.557404] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2456.309378] Key type lgssc unregistered [ 2456.626583] LNet: 84943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2456.634768] LNetError: 84943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2456.649909] LNet: Removed LNI 192.168.204.154@tcp [ 2457.278345] Key type .llcrypt unregistered [ 2457.280758] Key type ._llcrypt unregistered [ 2461.073912] Key type ._llcrypt registered [ 2461.078034] Key type .llcrypt registered [ 2461.187768] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2466.164708] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules [ 2467.527248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2467.565356] alg: No test for adler32 (adler32-zlib) [ 2468.704428] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 2468.927782] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2470.616063] Key type lgssc registered [ 2471.999595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2473.062812] LustreError: 86124:0:(lu_object.c:1434:lu_context_key_register()) cfs_race id 60d sleeping [ 2490.963605] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2494.172956] LustreError: 86124:0:(lu_object.c:1434:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2500.136706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2500.162781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2501.795294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.290681] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2511.350184] LustreError: 87134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.379058] LustreError: 87134:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2516.455304] LustreError: 87134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.460511] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:31:25 (1779975085) [ 2521.588918] LustreError: 87134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.510367] LustreError: 87148:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2522.523730] LustreError: 87148:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2522.539431] LustreError: 87148:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2524.921903] Lustre: server umount lustre-MDT0000 complete [ 2539.972179] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2543.121507] Key type lgssc unregistered [ 2543.548162] LNet: 88608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2543.565479] LNetError: 88608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2543.590324] LNet: Removed LNI 192.168.204.154@tcp [ 2544.597971] Key type .llcrypt unregistered [ 2544.604180] Key type ._llcrypt unregistered [ 2564.872630] Key type ._llcrypt registered [ 2564.875568] Key type .llcrypt registered [ 2565.011739] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2566.109832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2566.240822] alg: No test for adler32 (adler32-zlib) [ 2567.330848] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 2567.620494] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2569.303894] Key type lgssc registered [ 2570.823086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2578.223648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2578.254444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.840944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2583.833360] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2589.171268] LustreError: 89811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.193564] LustreError: 89811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2592.695228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.743581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2598.650199] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2605.375769] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2609.315947] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2616.260703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2616.968927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.095324] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2628.881923] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2680.538266] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:34:08 (1779975248) [ 2682.930906] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2685.664654] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:34:13 (1779975253) [ 2691.046717] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2691.049962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2691.081807] Lustre: Skipped 2 previous similar messages [ 2691.084613] Lustre: Skipped 1 previous similar message [ 2696.171755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2696.184281] Lustre: Skipped 3 previous similar messages [ 2697.256349] Lustre: server umount lustre-MDT0000 complete [ 2701.054614] LustreError: 89792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779975271 with bad export cookie 5882357656882181089 [ 2701.060916] LustreError: 89792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2701.062238] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2701.280133] LustreError: 89807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2701.284681] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2701.432545] Lustre: server umount lustre-MDT0001 complete [ 2707.551707] Lustre: server umount lustre-OST0000 complete [ 2722.030243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.875330] LustreError: 93750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.887996] LustreError: 93750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2723.101312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2728.427256] LustreError: 93749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.466461] LustreError: 93749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2728.991912] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2732.513283] LustreError: 93749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2737.701817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.570526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2743.948369] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2750.775132] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2755.079751] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2768.693756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2768.977257] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2768.989221] LustreError: 93738:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2769.035353] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2769.049757] LustreError: 95608:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 2769.057730] LustreError: 95608:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2769.067358] LustreError: 95608:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2769.139681] Lustre: server umount lustre-OST0000 complete [ 2769.153143] LustreError: 95608:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2778.281989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2778.517747] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2778.527451] LustreError: 93739:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2778.547921] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2778.561424] LustreError: 96070:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 2778.569322] LustreError: 96070:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2778.580351] LustreError: 96070:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2778.681601] Lustre: server umount lustre-OST0000 complete [ 2778.685439] LustreError: 96070:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2789.604113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2789.840448] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2789.844649] Lustre: Found index 0 for lustre-OST0000, updating log [ 2789.865635] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2789.965413] Lustre: lustre-OST0000: new disk, initializing [ 2789.972935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2790.252190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.164575] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2800.885227] Lustre: Failing over lustre-OST0000 [ 2801.144451] Lustre: server umount lustre-OST0000 complete [ 2807.779796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2807.780645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2807.803266] Lustre: Skipped 1 previous similar message [ 2813.923384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2813.930296] Lustre: Skipped 4 previous similar messages [ 2816.073437] Lustre: server umount lustre-MDT0000 complete [ 2819.041136] LustreError: 93749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2819.058071] LustreError: 93749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2820.168858] LustreError: 93729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779975391 with bad export cookie 5882357656882183833 [ 2820.172514] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2820.181823] LustreError: 93729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2820.602222] Lustre: server umount lustre-MDT0001 complete [ 2830.481190] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:36:38 (1779975398) [ 2850.068764] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2862.175138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2862.575978] LustreError: 98920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.727956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2867.150538] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2876.697772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.630993] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2885.728879] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2893.168181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2894.004847] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2900.888762] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2901.160556] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2901.172417] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2901.193685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2906.596776] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2906.610610] Lustre: Skipped 1 previous similar message [ 2906.619428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2906.627165] LustreError: Skipped 1 previous similar message [ 2907.642800] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 2907.681452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2907.682626] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2907.691943] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2907.698810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2909.454182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2909.533985] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2909.629889] Lustre: lustre-OST0001: new disk, initializing [ 2909.634611] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2909.966962] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2909.969819] Lustre: Skipped 2 previous similar messages [ 2914.964379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2914.983792] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2915.140492] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2916.149439] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2924.038264] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2928.351927] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2939.360810] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2939.371373] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.389353] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2944.581158] LustreError: 98914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2944.601182] LustreError: 98914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2944.710548] Lustre: server umount lustre-MDT0001 complete [ 2955.775772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.896915] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2955.902697] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2955.961304] Lustre: 98907:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2957.288641] LustreError: 102728:0:(obd_mount.c:212:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2957.298669] LustreError: 102728:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2957.312402] LustreError: 98912:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2957.402566] LustreError: 102738:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.204.154@tcp: cfg command failed: rc = -17 [ 2957.408991] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2957.408991] [ 2961.034919] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2961.916820] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2971.241836] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 09:38:59 (1779975539) [ 2990.907825] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3006.944259] Lustre: 104765:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3035.292839] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3041.788597] Lustre: 105900:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3145.696037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3145.708420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.721078] Lustre: Skipped 2 previous similar messages [ 3145.731272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3145.742222] Lustre: Skipped 4 previous similar messages [ 3151.265135] Lustre: server umount lustre-MDT0000 complete [ 3151.336605] LustreError: 98914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3151.349594] LustreError: 98914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3155.024119] LustreError: 98900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779975725 with bad export cookie 5882357656882184764 [ 3155.024526] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3155.038236] LustreError: 98900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3155.387559] Lustre: server umount lustre-MDT0001 complete [ 3170.146211] Lustre: server umount lustre-OST0000 complete [ 3172.319748] Lustre: 89122:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779975727/real 1779975727] req@ffff9b438c214700 x1866439210668032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779975743 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3172.350673] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3172.368561] Lustre: Skipped 3 previous similar messages [ 3173.965738] Lustre: server umount lustre-OST0001 complete [ 3178.829078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3188.520204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3197.318876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3206.821956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3227.442138] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3238.463631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3238.477669] Lustre: lustre-MDT0000: reset Object Index mappings [ 3239.123587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3239.131084] Lustre: Skipped 1 previous similar message [ 3243.583909] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3253.096101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.119703] Lustre: lustre-MDT0001: reset Object Index mappings [ 3258.517061] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3262.427101] Lustre: 110982:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3269.959579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3277.712964] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3279.010937] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3283.118826] LustreError: 111335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.137363] LustreError: 111335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3283.138188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3287.088850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3288.923660] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3288.929332] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3294.691849] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3304.119637] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3308.473857] Lustre: 112843:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3385.193500] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:45:52 (1779975952) [ 3410.404837] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.405710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3410.418588] Lustre: Skipped 2 previous similar messages [ 3410.436368] Lustre: Skipped 7 previous similar messages [ 3414.202704] Lustre: server umount lustre-MDT0000 complete [ 3418.590072] LustreError: 109850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779975989 with bad export cookie 5882357656882231874 [ 3418.599288] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3418.606023] LustreError: 109850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3425.222843] Lustre: server umount lustre-MDT0001 complete [ 3435.796276] Lustre: server umount lustre-OST0000 complete [ 3447.405996] Lustre: server umount lustre-OST0001 complete [ 3458.081885] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 3467.008846] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3504.223925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3504.608388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3504.643758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3504.720244] Lustre: lustre-MDT0000: new disk, initializing [ 3504.810066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3504.816523] Lustre: Skipped 3 previous similar messages [ 3504.858143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3510.257832] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3519.641806] Lustre: server umount lustre-MDT0000 complete [ 3531.335958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3531.674707] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3531.694341] Lustre: 117063:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3531.715464] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3537.364459] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3542.964238] Lustre: server umount lustre-MDT0000 complete [ 3551.965336] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:48:40 (1779976120) [ 3553.446906] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3555.371678] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:48:43 (1779976123) [ 3573.439694] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3583.142422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3588.271840] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3598.150293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3598.332327] Lustre: 119078:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3598.407860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3598.414275] Lustre: Skipped 1 previous similar message [ 3598.545232] Lustre: lustre-MDT0001: new disk, initializing [ 3598.725549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3598.737652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3604.223890] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3609.233683] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3615.857090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3616.165787] Lustre: lustre-OST0000: new disk, initializing [ 3616.169293] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3618.220045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3618.229192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3618.355670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3621.931208] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3633.050450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3633.219043] Lustre: lustre-OST0001: new disk, initializing [ 3633.222935] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3634.731046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3634.743860] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3634.879095] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3641.425284] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3653.672917] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3657.925469] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4110.038337] Lustre: 127890:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4110.045720] Lustre: 127890:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4110.060372] Lustre: 127890:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4110.066816] Lustre: 127890:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4110.076890] Lustre: 127890:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4110.081987] Lustre: 127890:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4452.333707] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:03:40 (1779977020) [ 4469.220154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4469.246352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.268719] Lustre: Skipped 3 previous similar messages [ 4469.275219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4469.286141] Lustre: Skipped 4 previous similar messages [ 4473.906609] Lustre: server umount lustre-MDT0000 complete [ 4476.387327] LustreError: 122265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4476.430965] LustreError: 122265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 4479.109792] LustreError: 119810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779977050 with bad export cookie 5882357656882303785 [ 4479.132361] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4479.141598] LustreError: 119810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4481.522060] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4481.538609] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4481.556161] Lustre: Skipped 4 previous similar messages [ 4481.556190] Lustre: Skipped 4 previous similar messages [ 4485.974965] Lustre: server umount lustre-MDT0001 complete [ 4497.420841] Lustre: server umount lustre-OST0000 complete [ 4516.319145] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4516.526133] Lustre: server umount lustre-OST0001 complete [ 4526.102701] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4538.037365] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4585.547566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4585.865607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4585.947204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4586.027582] Lustre: lustre-MDT0000: new disk, initializing [ 4586.140306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4586.150441] Lustre: Skipped 5 previous similar messages [ 4586.162965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4591.732821] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4604.824924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.921823] Lustre: 130935:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4604.997121] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4605.002477] Lustre: Skipped 1 previous similar message [ 4605.072428] Lustre: lustre-MDT0001: new disk, initializing [ 4605.229403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4605.240115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4611.922562] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4621.713103] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4627.767945] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4637.396075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4637.838987] Lustre: lustre-OST0000: new disk, initializing [ 4637.844527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4638.053938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4638.056902] Lustre: Skipped 1 previous similar message [ 4639.748222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4639.771555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4639.919585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4644.694915] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4653.958870] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4664.478874] Lustre: lustre-MDD0000: changelog on [ 4667.432292] Lustre: lustre-MDD0001: changelog on [ 7014.685469] Lustre: lustre-MDD0001: changelog off [ 7017.269457] Lustre: 130941:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 7018.550900] Lustre: lustre-MDD0000: changelog off [ 7025.633598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7025.656153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7025.688587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7025.708807] Lustre: Skipped 3 previous similar messages [ 7026.147939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7029.250825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7029.267143] Lustre: Skipped 2 previous similar messages [ 7031.592053] Lustre: server umount lustre-OST0000 complete [ 7039.470296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7039.477902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7039.485312] Lustre: Skipped 2 previous similar messages [ 7044.579020] LustreError: 130946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7044.693481] LustreError: 130946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7044.946081] Lustre: server umount lustre-MDT0000 complete [ 7049.704810] LustreError: 130946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7049.798287] LustreError: 130946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7050.891606] LustreError: 130926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779979621 with bad export cookie 5882357656883133320 [ 7050.921494] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7050.923819] LustreError: 130926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7052.049637] Lustre: server umount lustre-MDT0001 complete [ 7065.455977] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7069.157470] Key type lgssc unregistered [ 7069.530856] LNet: 141745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7069.538901] LNetError: 141745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7069.579892] LNet: Removed LNI 192.168.204.154@tcp [ 7070.771124] Key type .llcrypt unregistered [ 7070.772908] Key type ._llcrypt unregistered [ 7097.716390] Key type ._llcrypt registered [ 7097.717983] Key type .llcrypt registered [ 7097.916920] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 7114.126673] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7115.709974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7115.761019] alg: No test for adler32 (adler32-zlib) [ 7116.951346] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 7117.204127] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 7118.959697] Key type lgssc registered [ 7120.203966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7170.677101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7170.738109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7172.341708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7172.437602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7172.639672] Lustre: lustre-MDT0000: new disk, initializing [ 7172.888738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7172.955742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7178.416316] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7192.197612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7192.414965] Lustre: 144919:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7192.493761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7192.501583] Lustre: Skipped 1 previous similar message [ 7192.637490] Lustre: lustre-MDT0001: new disk, initializing [ 7192.817444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7192.872361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7192.888989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7199.491235] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7209.469762] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7214.584177] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7223.180741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7223.520529] Lustre: lustre-OST0000: new disk, initializing [ 7223.538580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7223.735166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7230.895627] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7232.087992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7232.101682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7232.266573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7238.613732] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7245.069724] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_module obdecho/obdecho [ 7246.684912] LustreError: 147539:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7246.691762] LustreError: 147539:0:(obd_config.c:880:class_setup()) setup ec failed (-22) [ 7248.714517] LustreError: 147635:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 7263.201241] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7263.207943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7263.219075] Lustre: Skipped 2 previous similar messages [ 7263.223073] Lustre: Skipped 1 previous similar message [ 7267.408321] Lustre: server umount lustre-MDT0000 complete [ 7271.880227] LustreError: 144910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779979842 with bad export cookie 234509615125981284 [ 7271.887967] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7271.898346] LustreError: 144910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7272.736992] Lustre: server umount lustre-MDT0001 complete [ 7288.604473] Lustre: server umount lustre-OST0000 complete [ 7315.591475] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 7329.164383] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7375.414370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7375.722966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7375.819733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7375.973556] Lustre: lustre-MDT0000: new disk, initializing [ 7376.208185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7376.262355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7381.825796] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7394.702651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7394.802250] Lustre: 151322:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7394.875649] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7394.881769] Lustre: Skipped 1 previous similar message [ 7394.997216] Lustre: lustre-MDT0001: new disk, initializing [ 7395.112548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7395.151443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7395.172981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7401.853411] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7411.179133] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7416.866413] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7425.204082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7425.503545] Lustre: lustre-OST0000: new disk, initializing [ 7425.518881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7425.676180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7432.576348] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7434.351017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7434.365202] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7434.472887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7441.218622] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7447.592116] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7447.924968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7453.805929] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7454.166527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7460.335535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7460.342162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7460.358273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7460.358312] LustreError: Skipped 1 previous similar message [ 7460.405528] Lustre: Skipped 1 previous similar message [ 7462.989942] Lustre: server umount lustre-OST0000 complete [ 7470.563350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7470.579270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7470.597107] Lustre: Skipped 1 previous similar message [ 7470.623481] Lustre: Skipped 2 previous similar messages [ 7475.681383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7475.688221] Lustre: Skipped 1 previous similar message [ 7480.801575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7480.810947] Lustre: Skipped 1 previous similar message [ 7482.335155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7482.748416] Lustre: server umount lustre-MDT0000 complete [ 7485.922837] LustreError: 151332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7485.949867] LustreError: 151332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7487.827105] LustreError: 151314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779980058 with bad export cookie 234509615125982243 [ 7487.833109] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7487.840072] LustreError: 151314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7488.403080] Lustre: server umount lustre-MDT0001 complete [ 7493.262378] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:54:20 (1779980060) [ 7495.565705] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7497.742950] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:54:25 (1779980065) [ 7507.749594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7508.298202] LustreError: 155332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7508.400762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7513.591800] LustreError: 155332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7515.888908] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7518.716330] LustreError: 155332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7524.838894] LustreError: 155331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7524.932544] LustreError: 155331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7532.890929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7541.582510] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7549.496570] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7554.916121] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7565.600077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7566.214058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7566.222533] Lustre: Skipped 1 previous similar message [ 7574.800898] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7582.760315] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7594.626360] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 7595.509623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7595.526047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7595.556817] LustreError: 155332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7595.584745] LustreError: 155332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7595.702446] Lustre: server umount lustre-MDT0000 complete [ 7600.911621] LustreError: 155311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779980171 with bad export cookie 234509615125983125 [ 7600.925382] LustreError: 155311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7600.925736] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7601.124870] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7601.137338] Lustre: Skipped 2 previous similar messages [ 7607.402614] Lustre: server umount lustre-MDT0001 complete [ 7614.754083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7631.517229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7631.692712] LustreError: 158843:0:(llog.c:1643:llog_backup()) MGC192.168.204.154@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7631.702546] Lustre: 158843:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.154@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7634.561688] LustreError: 158850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7634.601381] LustreError: 158850:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 7670.766200] LustreError: 158848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7670.806191] LustreError: 158848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 7692.256186] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7692.259628] Lustre: 158843:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.204.154@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7720.417259] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7720.427737] Lustre: 155322:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.204.154@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7733.266155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7736.901786] LustreError: 158848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7736.941140] LustreError: 158848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 43 previous similar messages [ 7738.362276] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7739.280843] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7740.900143] LustreError: 155322:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7749.616362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7760.869959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7770.109773] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3412540a1ff88ee [ 7770.127583] LustreError: 159581:0:(ldlm_resource.c:1170:ldlm_resource_complain()) MGC192.168.204.154@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000eb8cc80a) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7770.153100] Lustre: MGC192.168.204.154@tcp: Connection restored to 0@lo (at 0@lo) [ 7775.463834] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7775.735911] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7782.515896] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7784.622927] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7789.618624] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7791.424315] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7816.676738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7816.679346] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7816.690940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7816.710644] Lustre: Skipped 1 previous similar message [ 7816.723176] Lustre: Skipped 5 previous similar messages [ 7821.271644] Lustre: server umount lustre-MDT0000 complete [ 7825.235924] LustreError: 155312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779980396 with bad export cookie 234509615125989614 [ 7825.245422] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7825.253356] LustreError: 155312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7825.813940] Lustre: server umount lustre-MDT0001 complete [ 7839.385515] Lustre: server umount lustre-OST0000 complete [ 7862.845226] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 7872.964499] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7916.393329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7916.826146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7916.902170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7917.035575] Lustre: lustre-MDT0000: new disk, initializing [ 7917.120712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7917.129266] Lustre: Skipped 1 previous similar message [ 7917.186722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7922.934962] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7935.733719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7935.896291] Lustre: 163711:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7936.366402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7942.066774] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7951.287933] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7954.981244] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7961.623703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7961.881366] Lustre: lustre-OST0000: new disk, initializing [ 7961.883947] Lustre: Skipped 1 previous similar message [ 7961.888762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7961.898373] Lustre: Skipped 2 previous similar messages [ 7967.777395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7967.783544] Lustre: Skipped 1 previous similar message [ 7967.792597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7967.898939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7968.137266] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7975.263191] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7980.202959] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7980.455091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7985.818219] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7986.173582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7988.720164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7988.742748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7988.797847] Lustre: Skipped 1 previous similar message [ 7988.808492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7994.827458] Lustre: server umount lustre-OST0000 complete [ 8003.041486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8003.046200] LustreError: Skipped 1 previous similar message [ 8014.304562] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8014.466415] Lustre: server umount lustre-MDT0000 complete [ 8018.916939] LustreError: 163722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8018.952758] LustreError: 163722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 34 previous similar messages [ 8019.152701] LustreError: 163700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779980590 with bad export cookie 234509615126006442 [ 8019.161221] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8019.700941] Lustre: server umount lustre-MDT0001 complete [ 8028.494087] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:03:16 (1779980596) [ 8035.119572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8039.747831] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8049.796091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8054.664730] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8060.499358] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8064.125450] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8070.862516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8077.286788] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8083.388932] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8093.581028] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:04:21 (1779980661) [ 8101.345715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8101.349224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8101.364977] Lustre: Skipped 3 previous similar messages [ 8101.374786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8101.388665] Lustre: Skipped 9 previous similar messages [ 8103.804784] Lustre: server umount lustre-OST0000 complete [ 8113.589978] Lustre: server umount lustre-MDT0000 complete [ 8117.377780] LustreError: 167700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779980688 with bad export cookie 234509615126007324 [ 8117.381312] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8117.386552] LustreError: 167700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8117.631952] Lustre: server umount lustre-MDT0001 complete [ 8128.302737] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 8130.952243] Key type lgssc unregistered [ 8131.225300] LNet: 171091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8131.234587] LNetError: 171091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8131.251744] LNet: Removed LNI 192.168.204.154@tcp [ 8131.978174] Key type .llcrypt unregistered [ 8131.980539] Key type ._llcrypt unregistered [ 8153.744653] Key type ._llcrypt registered [ 8153.746599] Key type .llcrypt registered [ 8153.858667] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8155.273820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8155.502619] alg: No test for adler32 (adler32-zlib) [ 8156.550235] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 8156.833226] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8158.503226] Key type lgssc registered [ 8160.198794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8169.291737] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8169.302895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8184.991754] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8190.111931] LustreError: 172319:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 8195.295417] LustreError: 172319:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8195.314629] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8195.323580] LustreError: MGC192.168.204.154@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8195.357400] LustreError: 172319:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8195.377236] LustreError: 172319:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -22 [ 8195.391848] LustreError: 172319:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 8195.513075] Lustre: server umount lustre-OST0000 complete [ 8195.521134] LustreError: 172319:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8206.548641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8208.075948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8212.886690] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8217.587123] LustreError: 172819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8217.615887] LustreError: 172819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8222.689077] LustreError: 172819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8222.895487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8223.244726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8227.579247] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8234.098734] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8237.658080] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8245.103241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8245.266148] LustreError: 174409:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8245.274958] Lustre: 174409:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.204.154@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8245.429906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8251.063432] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8256.632919] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8259.042088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8259.049795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8259.065873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8263.138134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8263.141945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8263.160409] Lustre: Skipped 2 previous similar messages [ 8268.256858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8268.266305] Lustre: Skipped 2 previous similar messages [ 8273.380616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8273.385569] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 8273.394228] Lustre: Skipped 2 previous similar messages [ 8273.654382] Lustre: server umount lustre-MDT0000 complete [ 8277.109206] LustreError: 172799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779980848 with bad export cookie 1032584759503461699 [ 8277.120654] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8277.122558] LustreError: 172799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8277.416676] Lustre: server umount lustre-MDT0001 complete [ 8292.153507] Lustre: server umount lustre-OST0000 complete [ 8302.842840] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 8313.728283] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8353.796248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8354.163408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8354.261952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8354.454249] Lustre: lustre-MDT0000: new disk, initializing [ 8354.543556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8354.576452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8359.863879] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8371.171576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8371.321322] Lustre: 177793:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8371.404940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8371.417324] Lustre: Skipped 1 previous similar message [ 8371.518953] Lustre: lustre-MDT0001: new disk, initializing [ 8371.633526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8371.685863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8371.698707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8375.489291] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8383.128856] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8387.553701] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8394.479696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8394.744732] Lustre: lustre-OST0000: new disk, initializing [ 8394.751804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8394.832793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8401.077068] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8403.520796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8403.532105] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8403.623425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8407.971310] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8412.928585] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8413.158633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8418.230948] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8418.476593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8424.423443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8424.423862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8424.429443] LustreError: Skipped 1 previous similar message [ 8424.438040] Lustre: Skipped 1 previous similar message [ 8424.439359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8426.693704] Lustre: server umount lustre-OST0000 complete [ 8433.120888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8433.126666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8433.136305] Lustre: Skipped 1 previous similar message [ 8433.144188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8433.156037] Lustre: Skipped 1 previous similar message [ 8444.383180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8444.539691] Lustre: server umount lustre-MDT0000 complete [ 8444.897351] LustreError: 178934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8444.918594] LustreError: 178934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8447.924312] LustreError: 179537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779981018 with bad export cookie 1032584759503462420 [ 8447.929650] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8447.937490] LustreError: 179537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8448.167767] Lustre: server umount lustre-MDT0001 complete [ 8457.761722] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:10:25 (1779981025) [ 8475.251967] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 8478.174377] Key type lgssc unregistered [ 8478.490583] LNet: 182319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8478.503607] LNetError: 182319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8478.525762] LNet: Removed LNI 192.168.204.154@tcp [ 8479.350419] Key type .llcrypt unregistered [ 8479.360311] Key type ._llcrypt unregistered [ 8497.950074] Key type ._llcrypt registered [ 8497.957106] Key type .llcrypt registered [ 8498.102260] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8499.547450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8499.572918] alg: No test for adler32 (adler32-zlib) [ 8500.678477] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 8500.965372] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8502.711273] Key type lgssc registered [ 8504.052604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8512.152605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8512.175193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8513.641409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8517.886543] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8523.236541] LustreError: 183520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8523.261385] LustreError: 183520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8526.426216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8526.783795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8531.386401] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8537.036415] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8541.141336] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8549.368494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8549.698728] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8549.703251] LustreError: 185159:0:(llog.c:1684:llog_backup()) MGC192.168.204.154@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8549.711389] Lustre: 185159:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.204.154@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8549.856204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8556.669617] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8562.690238] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8567.777239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8567.782142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8567.792405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8569.317518] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8569.321588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8569.351657] Lustre: Skipped 2 previous similar messages [ 8571.450446] Lustre: server umount lustre-MDT0000 complete [ 8574.432520] LustreError: 183521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8574.465716] LustreError: 183521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8575.680513] LustreError: 183501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779981146 with bad export cookie 11391829108377677080 [ 8575.691716] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8575.700284] LustreError: 183501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8576.204805] Lustre: server umount lustre-MDT0001 complete [ 8590.986923] Lustre: server umount lustre-OST0000 complete [ 8600.934649] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 8611.639777] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8652.290274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8652.703841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8652.743682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8652.813724] Lustre: lustre-MDT0000: new disk, initializing [ 8652.898076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8652.915900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8658.047142] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8669.267081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8669.462283] Lustre: 188538:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8669.537748] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8669.541925] Lustre: Skipped 1 previous similar message [ 8669.552473] LustreError: 189303:0:(llog.c:1684:llog_backup()) MGC192.168.204.154@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8669.560062] LustreError: 189303:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 8669.566535] Lustre: 189303:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.154@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8669.592509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8669.596432] Lustre: Skipped 1 previous similar message [ 8669.709778] Lustre: lustre-MDT0001: new disk, initializing [ 8669.832078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8669.882443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8669.898713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8674.328956] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8681.941965] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8685.524682] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8693.005883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8693.297622] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8693.300772] Lustre: Skipped 1 previous similar message [ 8693.307993] LustreError: 190256:0:(llog.c:1684:llog_backup()) MGC192.168.204.154@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8693.321858] LustreError: 190256:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 8693.329461] Lustre: 190256:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.154@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8693.341988] Lustre: 190256:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8693.375597] Lustre: lustre-OST0000: new disk, initializing [ 8693.387720] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8693.525132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8699.864140] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8702.572774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8702.587978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8702.748971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8708.097980] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8713.013830] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8713.261567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8719.046958] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8719.232643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8723.427174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8723.428688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8723.435753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8723.439194] Lustre: Skipped 2 previous similar messages [ 8727.625213] Lustre: server umount lustre-OST0000 complete [ 8733.665395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8733.672346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8733.683379] Lustre: Skipped 1 previous similar message [ 8733.692291] Lustre: Skipped 2 previous similar messages [ 8737.629055] Lustre: server umount lustre-MDT0000 complete [ 8738.786265] LustreError: 188549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8738.798468] LustreError: 188549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8741.689149] LustreError: 188528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779981312 with bad export cookie 11391829108377677801 [ 8741.698296] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8741.711451] LustreError: 188528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8742.129425] Lustre: server umount lustre-MDT0001 complete [ 8750.186359] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:15:18 (1779981318) [ 8766.776650] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8776.611123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8776.925218] LustreError: 193245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8776.990351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8781.261731] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8782.311864] LustreError: 193245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8790.522118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8795.559682] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8798.550951] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8805.569219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8812.783231] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8822.216191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8822.407996] Lustre: lustre-OST0001: new disk, initializing [ 8822.420169] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8822.514840] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8822.518505] Lustre: Skipped 2 previous similar messages [ 8829.145633] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8829.549464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8829.565044] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8829.658280] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8841.712416] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8849.243420] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8856.499704] Lustre: DEBUG MARKER: ADD OST3 [ 8861.701337] loop: module loaded [ 8861.796200] loop0: detected capacity change from 0 to 204800000 [ 8862.029513] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8862.043266] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8862.055232] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8868.698858] loop0: detected capacity change from 0 to 204800000 [ 8869.348401] Lustre: DEBUG MARKER: STOP OST3 [ 8874.667772] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8874.932182] Lustre: lustre-OST0003: new disk, initializing [ 8874.946097] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8876.136112] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8876.310263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8876.326656] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8876.382465] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8878.177183] LustreError: 197606:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8878.843790] Lustre: Failing over lustre-OST0003 [ 8879.046699] Lustre: server umount lustre-OST0003 complete [ 8881.642652] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8881.649314] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8881.650592] LustreError: 194697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8881.650602] LustreError: 194697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8881.662968] Lustre: Skipped 1 previous similar message [ 8881.670057] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8891.105946] Lustre: DEBUG MARKER: START OST3 again [ 8899.529211] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8900.057612] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8900.092304] Lustre: Skipped 1 previous similar message [ 8900.119449] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8901.883354] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8902.105458] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8902.105818] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8908.516252] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8930.966628] Lustre: Failing over lustre-OST0003 [ 8931.102450] Lustre: server umount lustre-OST0003 complete [ 8932.833261] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8932.836728] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.853462] LustreError: Skipped 1 previous similar message [ 8932.867979] LustreError: 195872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8932.900325] LustreError: 195872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 8949.729363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8949.737828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8949.754579] Lustre: Skipped 1 previous similar message [ 8949.758545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8955.522544] Lustre: server umount lustre-MDT0000 complete [ 8959.502730] LustreError: 193225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779981530 with bad export cookie 11391829108377678683 [ 8959.508649] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8959.519239] LustreError: 193225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8961.519155] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8961.535942] Lustre: Skipped 5 previous similar messages [ 8966.121241] Lustre: server umount lustre-MDT0001 complete [ 8970.160542] Lustre: server umount lustre-OST0000 complete [ 8974.601446] Lustre: server umount lustre-OST0001 complete [ 8981.725868] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 8991.554392] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9030.866798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9031.126184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9031.157824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9031.297401] Lustre: lustre-MDT0000: new disk, initializing [ 9031.443278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9031.470702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9035.314458] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9044.920806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9045.093702] Lustre: 202884:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9045.396854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9049.854497] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9057.168334] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9061.225948] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9068.162150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9068.480166] Lustre: lustre-OST0000: new disk, initializing [ 9068.484354] Lustre: Skipped 1 previous similar message [ 9068.490834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9068.503797] Lustre: Skipped 2 previous similar messages [ 9069.647171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9069.655920] Lustre: Skipped 1 previous similar message [ 9069.659304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9069.728578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9074.327289] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9082.374489] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9086.708934] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9086.960528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9091.462401] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9091.682408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9095.657191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9095.657294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9095.674590] LustreError: Skipped 1 previous similar message [ 9095.678684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9095.690752] Lustre: Skipped 6 previous similar messages [ 9095.704982] Lustre: Skipped 2 previous similar messages [ 9100.016759] Lustre: server umount lustre-OST0000 complete [ 9118.175325] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9118.359453] Lustre: server umount lustre-MDT0000 complete [ 9120.228137] LustreError: 202890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9120.238802] LustreError: 202890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 9121.865368] LustreError: 204626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779981692 with bad export cookie 11391829108377680979 [ 9121.869885] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9121.874690] LustreError: 204626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9122.128868] Lustre: server umount lustre-MDT0001 complete [ 9126.194611] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:21:33 (1779981693) [ 9142.508079] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 9152.268719] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9191.700277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9191.937451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9191.977477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9192.079628] Lustre: lustre-MDT0000: new disk, initializing [ 9192.222896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9197.018660] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9207.759238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9207.912241] Lustre: 209057:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9208.184383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9212.230432] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9220.639883] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9225.060531] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9232.809482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9234.888663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9239.900867] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9248.401495] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9253.365120] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9253.623973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9259.303928] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9259.562660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9265.634325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9265.635357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9265.648232] LustreError: Skipped 1 previous similar message [ 9265.659632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9265.669625] Lustre: Skipped 3 previous similar messages [ 9265.701150] Lustre: Skipped 8 previous similar messages [ 9276.383514] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9276.731971] Lustre: server umount lustre-OST0000 complete [ 9287.813756] Lustre: server umount lustre-MDT0000 complete [ 9291.237125] LustreError: 209064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9293.272206] LustreError: 209050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779981864 with bad export cookie 11391829108377681854 [ 9293.282160] LustreError: 209050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9293.287950] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9293.707307] Lustre: server umount lustre-MDT0001 complete [ 9301.979318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9302.513498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9302.518042] Lustre: Skipped 5 previous similar messages [ 9306.961098] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9316.005470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9321.113613] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9327.323568] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9331.557729] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9338.157908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9343.720289] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9349.892535] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9363.435298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9363.444969] Lustre: Skipped 8 previous similar messages [ 9366.799523] Lustre: server umount lustre-OST0000 complete [ 9372.648663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9372.662289] LustreError: Skipped 1 previous similar message [ 9377.178235] Lustre: server umount lustre-MDT0000 complete [ 9381.259656] LustreError: 212854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779981952 with bad export cookie 11391829108377682743 [ 9381.268846] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9381.273492] LustreError: 212854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9381.691722] Lustre: server umount lustre-MDT0001 complete [ 9493.032469] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 9496.550736] Key type lgssc unregistered [ 9496.876522] LNet: 216185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9496.891481] LNetError: 216185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9496.918575] LNet: Removed LNI 192.168.204.154@tcp [ 9497.918202] Key type .llcrypt unregistered [ 9497.920437] Key type ._llcrypt unregistered [ 9531.122383] Key type ._llcrypt registered [ 9531.124250] Key type .llcrypt registered [ 9531.287330] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 9550.064804] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9551.380941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9551.421601] alg: No test for adler32 (adler32-zlib) [ 9552.512422] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 9552.850040] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 9554.527178] Key type lgssc registered [ 9555.857967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9599.237320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9599.266945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9600.685551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9600.732390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9600.843258] Lustre: lustre-MDT0000: new disk, initializing [ 9600.943430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9600.959742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9605.463934] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9616.919047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9617.153866] Lustre: 219509:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9617.239041] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9617.247203] Lustre: Skipped 1 previous similar message [ 9617.370381] Lustre: lustre-MDT0001: new disk, initializing [ 9617.488305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9617.526779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9617.538532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9622.369672] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9631.305898] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9635.579958] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9642.459591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9642.763951] Lustre: lustre-OST0000: new disk, initializing [ 9642.768379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9642.838331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9649.236041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9649.257347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9649.304847] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9649.380823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9656.622800] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9661.075533] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9661.296557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9665.327311] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9665.535497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9670.117825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9670.122915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9670.131809] Lustre: Skipped 1 previous similar message [ 9670.132068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9670.151308] Lustre: Skipped 1 previous similar message [ 9673.776889] Lustre: server umount lustre-OST0000 complete [ 9678.819740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9678.833225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9678.855193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9680.354115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9680.354787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9680.386098] Lustre: Skipped 1 previous similar message [ 9684.888859] Lustre: server umount lustre-MDT0000 complete [ 9689.686498] LustreError: 219500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779982260 with bad export cookie 16623138057697434034 [ 9689.696507] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9689.701450] LustreError: 219500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9690.179389] Lustre: server umount lustre-MDT0001 complete [ 9713.606248] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9725.301131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9725.871483] LustreError: 224037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9725.884976] LustreError: 224037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9725.949890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9730.466329] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9731.048056] LustreError: 224037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9735.139695] LustreError: 224037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9740.261100] LustreError: 224037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9740.431796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9740.968411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9746.410452] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9750.429987] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9758.627040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9759.111915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9766.679894] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9776.199026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9776.348730] Lustre: lustre-OST0001: new disk, initializing [ 9776.352849] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9776.461552] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9778.893442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9778.908528] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9779.010408] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9784.101566] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9797.703624] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9808.921897] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9829.502342] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:33:17 (1779982397) [ 9832.934028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9832.948287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9832.969312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9837.789742] Lustre: server umount lustre-MDT0000 complete [ 9840.611671] LustreError: 224033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9840.624675] LustreError: 224033:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9841.431081] LustreError: 224018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779982412 with bad export cookie 16623138057697434909 [ 9841.442445] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9841.444258] LustreError: 224018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9841.909117] Lustre: server umount lustre-MDT0001 complete [ 9856.661779] Lustre: server umount lustre-OST0000 complete [ 9861.904209] Lustre: 217731:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779982416/real 1779982416] req@ffff9b438803d880 x1866446534726784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779982432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9861.960524] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9861.978781] Lustre: Skipped 3 previous similar messages [ 9862.436691] Lustre: server umount lustre-OST0001 complete [ 9872.793571] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 9888.198183] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9938.892244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9939.403200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9939.468084] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9939.625891] Lustre: lustre-MDT0000: new disk, initializing [ 9939.822688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9939.867783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9945.958937] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9959.544721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9959.700371] Lustre: 230230:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9959.782633] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9959.786421] Lustre: Skipped 1 previous similar message [ 9959.886905] Lustre: lustre-MDT0001: new disk, initializing [ 9960.067155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9960.079065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9964.938653] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9973.491991] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9977.416211] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9985.585939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9985.875456] Lustre: lustre-OST0000: new disk, initializing [ 9985.883900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9987.446133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9987.461474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9987.555996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9993.508886] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10003.229846] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10010.321834] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10010.525932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10016.371988] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10016.615603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10023.403723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10023.422307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10023.442836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10023.460045] LustreError: Skipped 1 previous similar message [10023.489449] Lustre: Skipped 5 previous similar messages [10025.300472] Lustre: server umount lustre-OST0000 complete [10031.587278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10031.604853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10037.611891] Lustre: server umount lustre-MDT0000 complete [10038.758808] LustreError: 230237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10038.806585] LustreError: 230237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10042.931961] LustreError: 231008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779982613 with bad export cookie 16623138057697437051 [10042.944545] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10042.948055] LustreError: 231008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10043.275878] Lustre: server umount lustre-MDT0001 complete [10054.719767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10055.119082] LustreError: 234045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10055.203069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10055.210568] Lustre: Skipped 2 previous similar messages [10060.972842] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10071.687900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10076.462061] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10083.119749] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10088.287315] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10096.630742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10104.883668] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10111.722177] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10258.917016] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10258.931626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10258.942330] Lustre: Skipped 3 previous similar messages [10258.951253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10258.955016] Lustre: Skipped 2 previous similar messages [10262.869930] Lustre: server umount lustre-OST0000 complete [10273.334728] Lustre: server umount lustre-MDT0000 complete [10275.305847] LustreError: 236199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10275.317608] LustreError: 236199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10277.712968] LustreError: 234026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779982848 with bad export cookie 16623138057697437940 [10277.731522] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10278.082823] Lustre: server umount lustre-MDT0001 complete [10382.045721] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [10385.255904] Key type lgssc unregistered [10385.613353] LNet: 237299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10385.622811] LNetError: 237299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10385.642976] LNet: Removed LNI 192.168.204.154@tcp [10386.846308] Key type .llcrypt unregistered [10386.849821] Key type ._llcrypt unregistered [10416.465798] Key type ._llcrypt registered [10416.469029] Key type .llcrypt registered [10416.600966] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [10434.692671] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [10436.557634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10436.608795] alg: No test for adler32 (adler32-zlib) [10437.806709] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [10438.206970] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [10439.951173] Key type lgssc registered [10441.836941] Lustre: Echo OBD driver; http://www.lustre.org/ [10493.569246] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [10505.140220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10505.223522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10506.630923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10506.683503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10506.816501] Lustre: lustre-MDT0000: new disk, initializing [10506.935269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10506.962268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10511.835174] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10524.477090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10524.675361] Lustre: 241323:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10524.789367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10524.807188] Lustre: Skipped 1 previous similar message [10524.961953] Lustre: lustre-MDT0001: new disk, initializing [10525.051707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10525.096281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10525.108773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10529.750942] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10535.061351] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10541.732274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10542.076985] Lustre: lustre-OST0000: new disk, initializing [10542.087107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10542.215058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10546.796986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10546.809259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10546.981130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10549.401603] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10561.320440] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10561.548658] Lustre: lustre-OST0001: new disk, initializing [10561.556563] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10561.690320] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10568.350384] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10571.411718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10571.424713] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10571.545150] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10580.439648] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10588.346647] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10607.592646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10607.601363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10607.632793] Lustre: Skipped 1 previous similar message [10607.653936] Lustre: Skipped 2 previous similar messages [10612.194959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10612.201852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10612.211683] Lustre: Skipped 1 previous similar message [10612.221033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10613.820211] Lustre: server umount lustre-MDT0000 complete [10620.103542] LustreError: 241315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779983191 with bad export cookie 4373897102548225658 [10620.118244] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10620.123448] LustreError: 241315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10620.740944] Lustre: server umount lustre-MDT0001 complete [10636.237038] Lustre: server umount lustre-OST0000 complete [10638.303992] Lustre: 238846:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779983193/real 1779983193] req@ffff9b43880dc000 x1866447463725952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779983209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10638.350246] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10640.377871] Lustre: server umount lustre-OST0001 complete [10655.468586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10667.907851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10678.970463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10689.301214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10710.780236] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [10721.646473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10721.663918] Lustre: lustre-MDT0000: reset Object Index mappings [10722.077995] LustreError: 248753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10722.100309] LustreError: 248753:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10722.169672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10727.243492] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10727.394797] LustreError: 248753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10732.513611] LustreError: 248753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10736.522857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.559712] Lustre: lustre-MDT0001: reset Object Index mappings [10737.190342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10742.519960] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10746.141669] Lustre: 249859:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10753.941518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10754.426846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10761.477995] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10762.610940] LustreError: 250216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10767.721464] LustreError: 250215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10767.754948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10767.760634] LustreError: 250215:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10771.185336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10773.093510] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10778.933326] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10786.333582] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10789.268819] Lustre: 251705:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10800.310385] Lustre: server umount lustre-MDT0000 complete [10802.660940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10802.680855] Lustre: Skipped 3 previous similar messages [10802.688112] LustreError: 248747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10802.704566] LustreError: 248747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10805.913099] LustreError: 248732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779983376 with bad export cookie 4373897102548232448 [10805.918072] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10805.933358] LustreError: 248732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10806.300664] Lustre: server umount lustre-MDT0001 complete [10821.073987] Lustre: server umount lustre-OST0000 complete [10824.031417] Lustre: 238844:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779983378/real 1779983378] req@ffff9b4385252680 x1866447463779328/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779983394 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10824.056832] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10824.929816] Lustre: server umount lustre-OST0001 complete [10847.779745] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:50:15 (1779983415) [10864.829327] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [10874.662513] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [10914.522865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10914.887231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10914.941652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10915.060729] Lustre: lustre-MDT0000: new disk, initializing [10915.245934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10915.250207] Lustre: Skipped 1 previous similar message [10915.295892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10920.127641] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10933.074082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10933.208500] Lustre: 255929:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10933.278273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10933.284926] Lustre: Skipped 1 previous similar message [10933.383843] Lustre: lustre-MDT0001: new disk, initializing [10933.652282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10933.667797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10938.410523] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10946.789721] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10951.684611] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10959.113236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10959.376651] Lustre: lustre-OST0000: new disk, initializing [10959.383072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10961.460266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10961.467101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10961.600679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10966.024145] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10974.060192] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10980.034625] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10980.360703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10985.824475] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10986.161937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10992.622875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10992.625820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10992.638309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10992.638318] Lustre: Skipped 3 previous similar messages [10992.648551] Lustre: Skipped 1 previous similar message [10994.821512] Lustre: server umount lustre-OST0000 complete [11000.293654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11000.306557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11000.309467] Lustre: Skipped 1 previous similar message [11006.687501] Lustre: server umount lustre-MDT0000 complete [11007.980798] LustreError: 258581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11008.005988] LustreError: 258581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11011.068774] LustreError: 255921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779983582 with bad export cookie 4373897102548234534 [11011.069643] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11011.105536] LustreError: 255921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11011.613866] Lustre: server umount lustre-MDT0001 complete [11037.487363] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11052.231811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11053.390229] LustreError: 260464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11053.548025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11053.558812] Lustre: Skipped 2 previous similar messages [11060.077337] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11072.775698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11080.651765] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11084.647618] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11092.236467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11099.765409] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11109.639250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11109.879296] Lustre: lustre-OST0001: new disk, initializing [11109.885905] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11111.586500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11111.593909] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11111.798132] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11117.996874] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11128.756413] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11133.225610] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11144.448145] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11144.519310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11155.425279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11155.437802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11155.462884] Lustre: Skipped 2 previous similar messages [11155.468497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11155.471267] Lustre: Skipped 2 previous similar messages [11160.519257] Lustre: server umount lustre-MDT0000 complete [11163.110093] LustreError: 260459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11163.124579] LustreError: 260459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [11165.690644] LustreError: 262725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779983736 with bad export cookie 4373897102548235423 [11165.731503] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11166.183307] Lustre: server umount lustre-MDT0001 complete [11181.603640] Lustre: server umount lustre-OST0000 complete [11184.418226] Lustre: 238846:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779983739/real 1779983739] req@ffff9b4385250e00 x1866447463899136/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779983755 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11186.087803] Lustre: server umount lustre-OST0001 complete [11205.964576] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11218.866677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11219.590112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11219.598875] Lustre: Skipped 3 previous similar messages [11224.755186] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11235.280014] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11241.955524] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11247.378433] Lustre: 266861:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11259.629182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11264.554481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11268.457727] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11279.875795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11282.115348] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11288.779758] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11296.681614] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11300.903881] Lustre: 268711:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11311.619809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11311.668869] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11323.883950] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:58:11 (1779983891) [11341.795343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11341.804577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11341.829806] Lustre: Skipped 5 previous similar messages [11341.831327] Lustre: Skipped 6 previous similar messages [11346.178342] Lustre: server umount lustre-MDT0000 complete [11346.912652] LustreError: 266469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11346.948871] LustreError: 266469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [11350.439238] LustreError: 265731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779983921 with bad export cookie 4373897102548242213 [11350.453042] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11350.457082] LustreError: 265731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11356.861582] Lustre: server umount lustre-MDT0001 complete [11367.951550] Lustre: server umount lustre-OST0000 complete [11372.475869] Lustre: server umount lustre-OST0001 complete [11381.117226] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [11393.093728] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11453.062634] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11464.597684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11464.873412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11464.917827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11465.036319] Lustre: lustre-MDT0000: new disk, initializing [11465.177703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11469.879109] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11482.137933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11482.310196] Lustre: 272930:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11482.555334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11482.559978] Lustre: Skipped 4 previous similar messages [11482.592673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11488.089803] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11503.188285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11505.452526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11511.951721] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11524.789220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11526.644223] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11526.654412] Lustre: Skipped 1 previous similar message [11526.798166] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11533.130206] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11544.050232] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11547.993389] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11548.004873] Lustre: Skipped 1 previous similar message [11572.204644] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 12:02:19 (1779984139) [11577.832382] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11577.860141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11577.874826] Lustre: Skipped 6 previous similar messages [11577.886018] Lustre: Skipped 9 previous similar messages [11581.275838] Lustre: server umount lustre-MDT0000 complete [11585.315922] LustreError: 272921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779984156 with bad export cookie 4373897102548249619 [11585.326803] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11585.331171] LustreError: 272921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11585.638540] Lustre: server umount lustre-MDT0001 complete [11600.511644] Lustre: server umount lustre-OST0000 complete [11604.385684] Lustre: 238846:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779984159/real 1779984159] req@ffff9b43880dc000 x1866447464068224/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779984175 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11605.505426] Lustre: server umount lustre-OST0001 complete [11626.291305] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [11637.906996] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11686.243552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11686.675982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11686.716874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11686.724342] Lustre: Skipped 4 previous similar messages [11686.922685] Lustre: lustre-MDT0000: new disk, initializing [11686.939131] Lustre: Skipped 3 previous similar messages [11687.106210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11687.117183] Lustre: Skipped 3 previous similar messages [11692.966406] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11706.471167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11706.647654] Lustre: 279988:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11707.061282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11712.869838] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11721.628780] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11726.720547] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11735.841697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11738.617512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11743.963304] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11753.243566] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11755.978794] Lustre: Mounted lustre-client [11757.145461] LustreError: 282490:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9b438ae2d800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [11757.236544] Lustre: Unmounted lustre-client [11769.312213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11769.313964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11769.319902] LustreError: Skipped 1 previous similar message [11769.324335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11769.360984] Lustre: Skipped 2 previous similar messages [11775.408854] Lustre: server umount lustre-OST0000 complete [11785.973209] Lustre: server umount lustre-MDT0000 complete [11786.720369] LustreError: 279995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11786.756287] LustreError: 279995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [11790.710586] LustreError: 281734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779984361 with bad export cookie 4373897102548251152 [11790.712060] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11790.726082] LustreError: 281734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11791.266927] Lustre: server umount lustre-MDT0001 complete [11804.980484] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [11808.782229] Key type lgssc unregistered [11811.103890] LNet: 1 local NIs in recovery (showing 1): 192.168.204.154@tcp [11811.297711] LNet: 283771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11811.311808] LNetError: 283771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11888.172837] LNet: Removed LNI 192.168.204.154@tcp [11889.427209] Key type .llcrypt unregistered [11889.430084] Key type ._llcrypt unregistered [11912.727912] Key type ._llcrypt registered [11912.733277] Key type .llcrypt registered [11912.919139] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [11932.532376] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11934.278919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11934.340599] alg: No test for adler32 (adler32-zlib) [11935.571313] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [11935.994627] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [11937.839543] Key type lgssc registered [11939.255577] Lustre: Echo OBD driver; http://www.lustre.org/ [11981.176200] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:09:08 (1779984548) [11990.663695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11990.697952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11992.087779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11992.138515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11992.245834] Lustre: lustre-MDT0000: new disk, initializing [11992.448983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11992.483462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11999.259821] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12012.991090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12013.241255] Lustre: 286960:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12013.324431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12013.332603] Lustre: Skipped 1 previous similar message [12013.459843] Lustre: lustre-MDT0001: new disk, initializing [12013.710187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12013.778810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12013.813669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12020.065114] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12030.885450] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12035.578431] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12043.717134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12044.092485] Lustre: lustre-OST0000: new disk, initializing [12044.095949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12044.222466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12051.197807] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12053.109421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12053.130493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12053.289888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12059.769254] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12073.961339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12073.964832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12073.970293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12073.982371] Lustre: Skipped 1 previous similar message [12074.009198] Lustre: Skipped 1 previous similar message [12078.536141] Lustre: server umount lustre-OST0000 complete [12085.217035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12085.231233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12085.281472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12089.313167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12089.329668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12089.340072] Lustre: Skipped 1 previous similar message [12091.061749] Lustre: server umount lustre-MDT0000 complete [12096.867547] LustreError: 286950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779984667 with bad export cookie 2454923461193727493 [12096.869847] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12096.892556] LustreError: 286950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12097.864423] Lustre: server umount lustre-MDT0001 complete [12111.068863] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12115.685943] Key type lgssc unregistered [12116.004947] LNet: 290336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12116.016797] LNetError: 290336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12116.069770] LNet: Removed LNI 192.168.204.154@tcp [12117.559868] Key type .llcrypt unregistered [12117.561809] Key type ._llcrypt unregistered [12147.439957] Key type ._llcrypt registered [12147.447940] Key type .llcrypt registered [12147.563443] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [12148.801773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12148.857251] alg: No test for adler32 (adler32-zlib) [12149.987347] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [12150.342483] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [12152.151181] Key type lgssc registered [12153.513954] Lustre: Echo OBD driver; http://www.lustre.org/ [12162.459822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12162.500025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12164.165259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12169.161407] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12174.305950] LustreError: 291737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12174.332662] LustreError: 291737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12179.258843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12179.448152] LustreError: 291737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12179.816586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12185.113714] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12191.873774] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12196.585417] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12204.938530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12205.393376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12213.000469] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12220.351129] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12233.100107] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12235.673513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12243.614259] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [12245.482064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.500212] Lustre: Skipped 1 previous similar message [12245.500274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12245.526528] Lustre: Skipped 2 previous similar messages [12248.654848] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [12249.292966] Lustre: server umount lustre-MDT0000 complete [12250.698456] LustreError: 291737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12250.751413] LustreError: 291737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12253.774078] LustreError: 291732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12254.902722] LustreError: 291719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779984825 with bad export cookie 15200630260888124267 [12254.906250] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12254.914177] LustreError: 291719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12255.718508] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12255.732385] Lustre: Skipped 1 previous similar message [12255.738189] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12258.878272] LustreError: 291734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12258.899937] LustreError: 291734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12259.750286] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12259.752842] Lustre: Skipped 1 previous similar message [12261.620319] Lustre: server umount lustre-MDT0001 complete [12267.309257] Lustre: server umount lustre-OST0000 complete [12279.742370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12285.724872] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12290.003294] Lustre: 295475:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.154@tcp [12292.968052] Lustre: server umount MGS complete [12305.993437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12307.183689] LustreError: 296048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12307.218681] LustreError: 296048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12307.356434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12315.223410] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12324.028932] LustreError: 296044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12324.076486] LustreError: 296044:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12326.123475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12326.638483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12331.648802] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12341.900226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12342.454632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12350.162568] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12357.120800] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12359.090600] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12363.494625] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12365.058120] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12370.919391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12370.930478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12370.949556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12372.452254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12372.622899] Lustre: server umount lustre-OST0000 complete [12378.090353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12378.108284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12378.117631] Lustre: Skipped 1 previous similar message [12383.717461] LustreError: 296047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12383.742665] LustreError: 296047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12383.846269] Lustre: server umount lustre-MDT0000 complete [12389.074100] LustreError: 296029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779984960 with bad export cookie 15200630260888125107 [12389.091331] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12389.091688] LustreError: 296029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12389.692502] Lustre: server umount lustre-MDT0001 complete [12404.346516] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12408.344959] Key type lgssc unregistered [12408.857033] LNet: 299228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12408.878075] LNetError: 299228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12408.895922] LNet: Removed LNI 192.168.204.154@tcp [12410.267126] Key type .llcrypt unregistered [12410.270329] Key type ._llcrypt unregistered [12434.301728] Key type ._llcrypt registered [12434.310461] Key type .llcrypt registered [12434.463698] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [12452.486322] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [12454.106578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12454.162184] alg: No test for adler32 (adler32-zlib) [12455.325207] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [12455.603775] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [12457.319177] Key type lgssc registered [12458.904711] Lustre: Echo OBD driver; http://www.lustre.org/ [12501.419558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12501.439869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12502.960833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12503.027754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12503.166728] Lustre: lustre-MDT0000: new disk, initializing [12503.367319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12503.401926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12509.055541] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12520.086832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12520.262426] Lustre: 302355:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12520.336892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12520.342247] Lustre: Skipped 1 previous similar message [12520.463108] Lustre: lustre-MDT0001: new disk, initializing [12520.568021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12520.613576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12520.622972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12525.861630] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12535.070478] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12540.102283] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12550.815533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12551.299560] Lustre: lustre-OST0000: new disk, initializing [12551.303570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12551.399085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12560.537634] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12560.997111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12561.018427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12561.161815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12569.371897] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12575.693187] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12576.012770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12581.128756] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12581.317915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12586.978312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12586.980752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12586.986030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12587.006118] LustreError: Skipped 1 previous similar message [12587.041928] Lustre: Skipped 1 previous similar message [12589.982710] Lustre: server umount lustre-OST0000 complete [12597.218707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12597.221441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12597.259072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12597.269813] Lustre: Skipped 1 previous similar message [12600.478340] Lustre: server umount lustre-MDT0000 complete [12604.813757] LustreError: 304095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779985175 with bad export cookie 13280925765550587029 [12604.819323] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12604.824865] LustreError: 304095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12605.372231] Lustre: server umount lustre-MDT0001 complete [12616.603633] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 12:19:44 (1779985184) [12677.033832] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 12:20:44 (1779985244) [12732.884241] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12737.402171] Key type lgssc unregistered [12737.647251] LNet: 308959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12737.656264] LNetError: 308959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12737.668405] LNet: Removed LNI 192.168.204.154@tcp [12738.687789] Key type .llcrypt unregistered [12738.691650] Key type ._llcrypt unregistered [12859.894265] Key type ._llcrypt registered [12859.896823] Key type .llcrypt registered [12860.206449] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12860.845434] Key type .llcrypt unregistered [12860.848303] Key type ._llcrypt unregistered [12899.257720] Key type ._llcrypt registered [12899.260129] Key type .llcrypt registered [12899.346933] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12899.708684] Key type .llcrypt unregistered [12899.713730] Key type ._llcrypt unregistered [12937.711774] Key type ._llcrypt registered [12937.713448] Key type .llcrypt registered [12937.987266] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12938.309404] Key type .llcrypt unregistered [12938.311075] Key type ._llcrypt unregistered [13031.839653] Key type ._llcrypt registered [13031.842258] Key type .llcrypt registered [13031.933820] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13032.210275] Key type .llcrypt unregistered [13032.214187] Key type ._llcrypt unregistered [13066.094593] Key type ._llcrypt registered [13066.098311] Key type .llcrypt registered [13066.251334] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13066.711407] Key type .llcrypt unregistered [13066.713263] Key type ._llcrypt unregistered [13103.721632] Key type ._llcrypt registered [13103.725714] Key type .llcrypt registered [13103.909840] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13104.209648] Key type .llcrypt unregistered [13104.213815] Key type ._llcrypt unregistered [13143.966893] Key type ._llcrypt registered [13143.968413] Key type .llcrypt registered [13144.074346] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13144.510804] Key type .llcrypt unregistered [13144.514392] Key type ._llcrypt unregistered [13181.094260] Key type ._llcrypt registered [13181.096578] Key type .llcrypt registered [13181.254234] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13181.723094] Key type .llcrypt unregistered [13181.726084] Key type ._llcrypt unregistered [13226.679962] Key type ._llcrypt registered [13226.692689] Key type .llcrypt registered [13226.814378] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13227.219422] Key type .llcrypt unregistered [13227.222085] Key type ._llcrypt unregistered [13303.523256] Key type ._llcrypt registered [13303.525285] Key type .llcrypt registered [13303.747799] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13304.339100] Key type .llcrypt unregistered [13304.342946] Key type ._llcrypt unregistered [13344.148387] Key type ._llcrypt registered [13344.150659] Key type .llcrypt registered [13344.299982] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13344.588420] Key type .llcrypt unregistered [13344.590403] Key type ._llcrypt unregistered [13377.923099] Key type ._llcrypt registered [13377.924673] Key type .llcrypt registered [13378.026486] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13378.322276] Key type .llcrypt unregistered [13378.324319] Key type ._llcrypt unregistered [13444.684345] Key type ._llcrypt registered [13444.688688] Key type .llcrypt registered [13444.853200] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [13461.917256] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [13463.065462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13463.222984] alg: No test for adler32 (adler32-zlib) [13464.378652] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [13464.887853] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [13466.695189] Key type lgssc registered [13468.116782] Lustre: Echo OBD driver; http://www.lustre.org/ [13502.990794] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13212 sec ========= 12:34:30 (1779986070) [13505.293603] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:34:32 (1779986072) === [13510.269078] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:34:37 (1779986077) === [13561.129515] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13565.558308] Key type lgssc unregistered [13566.010566] LNet: 328213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13566.021291] LNetError: 328213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13566.040371] LNet: Removed LNI 192.168.204.154@tcp [13567.097781] Key type .llcrypt unregistered [13567.099552] Key type ._llcrypt unregistered