[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 469326785 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002315] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008686] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011138] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014048] *** VALIDATE selinux *** [ 0.022577] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026653] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029148] *** VALIDATE tmpfs *** [ 0.031289] *** VALIDATE proc *** [ 0.032260] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.035075] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037179] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043754] debug: unmapping init [mem 0xffffffff8b659000-0xffffffff8b660fff] [ 0.045461] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047233] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048040] ... version: 2 [ 0.049020] ... bit width: 48 [ 0.050023] ... generic registers: 4 [ 0.051037] ... value mask: 0000ffffffffffff [ 0.052020] ... max period: 00007fffffffffff [ 0.053024] ... fixed-purpose events: 3 [ 0.054017] ... event mask: 000000070000000f [ 0.056322] rcu: Hierarchical SRCU implementation. [ 0.058856] smp: Bringing up secondary CPUs ... [ 0.059746] x86: Booting SMP configuration: [ 0.060042] .... node #0, CPUs: #1 #2 #3 [ 0.064019] smp: Brought up 1 node, 4 CPUs [ 0.066029] smpboot: Max logical packages: 1 [ 0.067029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.203525] node 0 deferred pages initialised in 134ms [ 0.207009] devtmpfs: initialized [ 0.208204] x86/mm: Memory block size: 128MB [ 0.210762] gcov: version magic: 0x41383552 [ 0.212292] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.213105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.214370] pinctrl core: initialized pinctrl subsystem [ 0.215365] [ 0.215763] ************************************************************* [ 0.216016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.217019] ** ** [ 0.218021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.219015] ** ** [ 0.220012] ** This means that this kernel is built to expose internal ** [ 0.221017] ** IOMMU data structures, which may compromise security on ** [ 0.222020] ** your system. ** [ 0.223013] ** ** [ 0.224020] ** If you see this message and you are not debugging the ** [ 0.225021] ** kernel, report this immediately to your vendor! ** [ 0.226016] ** ** [ 0.227020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.228019] ************************************************************* [ 0.229745] NET: Registered protocol family 16 [ 0.230398] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.231104] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.232088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.233874] cpuidle: using governor menu [ 0.236319] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.239806] PCI: Using configuration type 1 for base access [ 0.242151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.250304] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.251028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.253211] cryptd: max_cpu_qlen set to 1000 [ 0.255239] ACPI: Added _OSI(Module Device) [ 0.257021] ACPI: Added _OSI(Processor Device) [ 0.259021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.261025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.266825] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.276444] ACPI: Interpreter enabled [ 0.277081] ACPI: PM: (supports S0 S3 S4 S5) [ 0.279015] ACPI: Using IOAPIC for interrupt routing [ 0.280972] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.283416] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.292618] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.294063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.296030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.299120] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.302184] acpiphp: Slot [2] registered [ 0.303177] acpiphp: Slot [5] registered [ 0.304169] acpiphp: Slot [6] registered [ 0.305117] acpiphp: Slot [7] registered [ 0.306135] acpiphp: Slot [8] registered [ 0.308195] acpiphp: Slot [9] registered [ 0.309184] acpiphp: Slot [10] registered [ 0.311259] acpiphp: Slot [3] registered [ 0.313115] acpiphp: Slot [4] registered [ 0.314106] acpiphp: Slot [11] registered [ 0.316164] acpiphp: Slot [12] registered [ 0.317126] acpiphp: Slot [13] registered [ 0.319148] acpiphp: Slot [14] registered [ 0.321157] acpiphp: Slot [15] registered [ 0.323309] acpiphp: Slot [16] registered [ 0.325287] acpiphp: Slot [17] registered [ 0.327281] acpiphp: Slot [18] registered [ 0.329129] acpiphp: Slot [19] registered [ 0.330099] acpiphp: Slot [20] registered [ 0.331150] acpiphp: Slot [21] registered [ 0.332123] acpiphp: Slot [22] registered [ 0.333091] acpiphp: Slot [23] registered [ 0.334087] acpiphp: Slot [24] registered [ 0.335136] acpiphp: Slot [25] registered [ 0.337180] acpiphp: Slot [26] registered [ 0.338153] acpiphp: Slot [27] registered [ 0.339123] acpiphp: Slot [28] registered [ 0.341177] acpiphp: Slot [29] registered [ 0.343158] acpiphp: Slot [30] registered [ 0.344154] acpiphp: Slot [31] registered [ 0.345218] PCI host bridge to bus 0000:00 [ 0.347024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.348028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.350029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.353033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.355031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.357031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.358200] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.360963] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.364619] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.374018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.379051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.382021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.383048] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.385027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.388139] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.390606] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.392184] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.397084] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.402023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.413024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.418029] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.422438] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.430027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.436023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.453032] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.462694] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.470022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.477024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.491028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.503235] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.511019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.515021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.530023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.538517] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.546022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.551025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.574024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.591532] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.598555] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.604035] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.625198] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.636868] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.642021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.660028] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.672020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.684139] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.686829] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.687450] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.689387] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.692190] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.696029] iommu: Default domain type: Passthrough [ 0.699556] SCSI subsystem initialized [ 0.702130] ACPI: bus type USB registered [ 0.704193] usbcore: registered new interface driver usbfs [ 0.706134] usbcore: registered new interface driver hub [ 0.708126] usbcore: registered new device driver usb [ 0.710196] pps_core: LinuxPPS API ver. 1 registered [ 0.712118] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.715091] PTP clock support registered [ 0.718060] EDAC MC: Ver: 3.0.0 [ 0.721277] PCI: Using ACPI for IRQ routing [ 0.724451] NetLabel: Initializing [ 0.726020] NetLabel: domain hash size = 128 [ 0.729029] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.731114] NetLabel: unlabeled traffic allowed by default [ 0.734055] vgaarb: loaded [ 0.736325] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.737015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.745688] clocksource: Switched to clocksource kvm-clock [ 0.863779] VFS: Disk quotas dquot_6.6.0 [ 0.865852] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.869448] *** VALIDATE ramfs *** [ 0.871550] *** VALIDATE hugetlbfs *** [ 0.875307] pnp: PnP ACPI init [ 0.878939] pnp: PnP ACPI: found 6 devices [ 0.895465] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.899119] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.901503] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.903917] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.906637] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.909268] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.912391] NET: Registered protocol family 2 [ 0.914765] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.919913] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.923405] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.928358] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.932046] TCP: Hash tables configured (established 65536 bind 65536) [ 0.935361] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.938849] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.942077] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.945660] NET: Registered protocol family 1 [ 0.948288] RPC: Registered named UNIX socket transport module. [ 0.950238] RPC: Registered udp transport module. [ 0.953157] RPC: Registered tcp transport module. [ 0.955495] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.958577] NET: Registered protocol family 44 [ 0.959696] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.962071] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.963684] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.965449] PCI: CLS 0 bytes, default 64 [ 0.966932] Unpacking initramfs... [ 2.385292] debug: unmapping init [mem 0xffff8d917cc54000-0xffff8d917ffbffff] [ 2.390251] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.392587] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.395358] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.897429] Initialise system trusted keyrings [ 2.898964] Key type blacklist registered [ 2.900849] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.910974] zbud: loaded [ 2.914539] *** VALIDATE nfs *** [ 2.916294] *** VALIDATE nfs4 *** [ 2.917772] pstore: using deflate compression [ 2.922626] Platform Keyring initialized [ 3.030347] NET: Registered protocol family 38 [ 3.032972] Key type asymmetric registered [ 3.034566] Asymmetric key parser 'x509' registered [ 3.036616] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.039964] io scheduler mq-deadline registered [ 3.041854] io scheduler kyber registered [ 3.043550] io scheduler bfq registered [ 3.049321] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.052093] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.055427] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.062442] ACPI: Power Button [PWRF] [ 3.067986] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.074475] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.087202] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.093950] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.110693] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.138227] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.173485] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.185051] Non-volatile memory driver v1.3 [ 3.186312] Linux agpgart interface v0.103 [ 3.215924] virtio_blk virtio1: [vda] 134736 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.219995] vda: detected capacity change from 0 to 68984832 [ 3.237393] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.240226] vdb: detected capacity change from 0 to 1073741824 [ 3.255244] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.258461] vdc: detected capacity change from 0 to 2621440000 [ 3.274931] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.277929] vdd: detected capacity change from 0 to 2621440000 [ 3.293296] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.296691] vde: detected capacity change from 0 to 4294967296 [ 3.313131] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.316722] vdf: detected capacity change from 0 to 4294967296 [ 3.322514] libphy: Fixed MDIO Bus: probed [ 3.328316] usbcore: registered new interface driver usbserial_generic [ 3.329960] usbserial: USB Serial support registered for generic [ 3.332243] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.335958] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.337413] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.339409] mousedev: PS/2 mouse device common for all mice [ 3.343030] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.348528] rtc_cmos 00:05: RTC can wake from S4 [ 3.351142] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.353398] rtc_cmos 00:05: registered as rtc0 [ 3.356820] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.359225] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.361568] intel_pstate: CPU model not supported [ 3.365701] hid: raw HID events driver (C) Jiri Kosina [ 3.367350] usbcore: registered new interface driver usbhid [ 3.368996] usbhid: USB HID core driver [ 3.370198] drop_monitor: Initializing network drop monitor service [ 3.372230] Initializing XFRM netlink socket [ 3.373563] NET: Registered protocol family 10 [ 3.375630] Segment Routing with IPv6 [ 3.376936] NET: Registered protocol family 17 [ 3.379051] mpls_gso: MPLS GSO support [ 3.385479] RAS: Correctable Errors collector initialized. [ 3.387031] AVX version of gcm_enc/dec engaged. [ 3.388926] AES CTR mode by8 optimization enabled [ 3.465689] sched_clock: Marking stable (3465646668, 0)->(4407126918, -941480250) [ 3.469373] registered taskstats version 1 [ 3.472646] Loading compiled-in X.509 certificates [ 3.475400] zswap: loaded using pool lzo/zbud [ 3.502392] Key type big_key registered [ 3.516701] Key type encrypted registered [ 3.518568] ima: No TPM chip found, activating TPM-bypass! [ 3.521436] ima: Allocated hash algorithm: sha1 [ 3.523356] ima: No architecture policies found [ 3.525448] evm: Initialising EVM extended attributes: [ 3.527623] evm: security.selinux [ 3.528977] evm: security.ima [ 3.530448] evm: security.capability [ 3.531929] evm: HMAC attrs: 0x1 [ 3.534844] rtc_cmos 00:05: setting system clock to 2026-02-18 07:24:18 UTC (1771399458) [ 3.541758] debug: unmapping init [mem 0xffffffff8c603000-0xffffffff8c7fffff] [ 3.544992] debug: unmapping init [mem 0xffffffff8b382000-0xffffffff8b658fff] [ 3.553088] Write protecting the kernel read-only data: 28672k [ 3.556635] debug: unmapping init [mem 0xffffffff89a03000-0xffffffff89bfffff] [ 3.559802] debug: unmapping init [mem 0xffffffff8a314000-0xffffffff8a3fffff] [ 3.595793] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.605949] systemd[1]: Detected virtualization kvm. [ 3.607602] systemd[1]: Detected architecture x86-64. [ 3.610057] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.639883] systemd[1]: No hostname configured. [ 3.641874] systemd[1]: Set hostname to . [ 3.644284] random: systemd: uninitialized urandom read (16 bytes read) [ 3.646911] systemd[1]: Initializing machine ID from random generator. [ 3.706814] random: ln: uninitialized urandom read (6 bytes read) [ 3.822685] random: systemd: uninitialized urandom read (16 bytes read) [ 3.825893] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.832065] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.836455] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.447609] device-mapper: uevent: version 1.0.3 [ 4.450190] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.304339] random: fast init done [ 5.507631] virtio_net virtio0 ens2: renamed from eth0 [ 5.682259] scsi host0: ata_piix [ 5.704570] scsi host1: ata_piix [ 5.709981] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.722956] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.580793] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.117275] random: crng init done [ 10.118819] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.781615] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.010182] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.256084] SELinux: Disabled at runtime. [ 12.318806] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.329471] systemd[1]: Detected virtualization kvm. [ 12.331295] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.003453] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.006574] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.012210] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.016923] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.020018] systemd[1]: Stopped Journal Service. [ 13.021019] hrtimer: interrupt took 3010870 ns [ OK ] Stopped Journal Service. [ 13.028337] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.034809] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ 13.244137] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.563629] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.962355] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.972647] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.075781] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.089726] EDAC sbridge: Ver: 1.1.2 [ 16.155582] Key type dns_resolver registered [ 16.552552] NFS: Registering the id_resolver key type [ 16.554592] Key type id_resolver registered [ 16.556352] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg314-server login: [ 76.604269] libcfs: loading out-of-tree module taints kernel. [ 76.722159] Key type ._llcrypt registered [ 76.725726] Key type .llcrypt registered [ 76.838691] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 98.368942] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 100.346307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 100.362571] alg: No test for adler32 (adler32-zlib) [ 101.852575] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 102.715750] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 104.407174] Key type lgssc registered [ 105.974740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 125.425489] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 162.690524] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 175.679986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 175.709996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.090987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 177.146043] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 177.309391] Lustre: lustre-MDT0000: new disk, initializing [ 177.391631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.408417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 181.501219] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.445107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.553971] Lustre: 6506:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 194.590361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 194.596483] Lustre: Skipped 1 previous similar message [ 194.675221] Lustre: lustre-MDT0001: new disk, initializing [ 194.732266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 194.782185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 194.803635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 200.117585] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.935493] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 214.691695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.045509] Lustre: lustre-OST0000: new disk, initializing [ 215.052371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.159729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.018099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 219.026139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 219.099033] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 221.640702] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.208928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.352167] Lustre: lustre-OST0001: new disk, initializing [ 237.356142] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.428678] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.676694] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.361307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 247.373758] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 247.466099] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 255.928703] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 263.510348] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 275.780554] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing check_logdir /tmp/testlogs/ [ 282.375348] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing yml_node [ 286.920574] Lustre: DEBUG MARKER: Client: 2.17.50.152 [ 289.359682] Lustre: DEBUG MARKER: MDS: 2.17.50.152 [ 291.760330] Lustre: DEBUG MARKER: OSS: 2.17.50.152 [ 293.298098] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Feb 18 02:29:07 EST 2026 [ 312.696722] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 314.457222] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 333.279926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 333.290500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.303683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 334.816505] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.819485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 334.842672] Lustre: Skipped 2 previous similar messages [ 334.881660] Lustre: Skipped 3 previous similar messages [ 336.287511] Lustre: server umount lustre-MDT0000 complete [ 340.516252] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771399795 with bad export cookie 9416951421212743043 [ 340.527909] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.545978] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 345.059086] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.061141] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.068817] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 345.094187] Lustre: Skipped 1 previous similar message [ 345.145501] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 347.125661] Lustre: server umount lustre-MDT0001 complete [ 351.288864] Lustre: server umount lustre-OST0000 complete [ 355.638339] Lustre: server umount lustre-OST0001 complete [ 363.070801] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 377.545043] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 419.199951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.743851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 419.885743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 420.045759] Lustre: lustre-MDT0000: new disk, initializing [ 420.341358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 420.397720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 426.192984] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 438.957878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 439.130277] Lustre: 15000:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 439.210748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 439.213901] Lustre: Skipped 1 previous similar message [ 439.337763] Lustre: lustre-MDT0001: new disk, initializing [ 439.515742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 439.625418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 439.643245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 445.071525] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 453.155711] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 457.480693] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 465.065178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.535145] Lustre: lustre-OST0000: new disk, initializing [ 465.539566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 465.646396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.373214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 467.379751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 467.548252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 471.858771] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 479.648996] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 484.623525] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 484.905122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 490.575288] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 490.926930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 498.145364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.150777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 498.204793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 498.223395] Lustre: Skipped 2 previous similar messages [ 499.746191] Lustre: server umount lustre-OST0000 complete [ 506.336293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 506.353173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.385960] Lustre: Skipped 1 previous similar message [ 506.400901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 518.633205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 518.641549] Lustre: Skipped 5 previous similar messages [ 519.135145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 519.458375] Lustre: server umount lustre-MDT0000 complete [ 523.715046] LustreError: 16743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771399978 with bad export cookie 9416951421212744576 [ 523.726183] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 523.736092] LustreError: 16743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 524.420783] Lustre: server umount lustre-MDT0001 complete [ 531.747926] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 02:33:05 (1771399985) [ 553.786847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.367133] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.482421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 559.413632] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 559.603963] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.708423] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.453755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.989274] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 581.373552] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 585.597428] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 588.154204] Lustre: Failing over lustre-MDT0000 [ 588.515644] Lustre: server umount lustre-MDT0000 complete [ 589.793742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.837806] Lustre: Skipped 1 previous similar message [ 589.858094] LustreError: 19468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 589.889350] LustreError: 19468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 601.057132] LustreError: 19467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.108185] LustreError: 19467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 605.176658] Lustre: 3661:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771400044/real 1771400044] req@ffff8d91f10dce00 x1857447062043520/t0(0) o400->MGC192.168.203.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771400060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 605.230321] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 610.011203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 615.891765] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 621.032561] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 621.036112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 621.089775] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 621.625538] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 631.698822] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 636.385726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 636.390652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.414641] Lustre: Skipped 1 previous similar message [ 636.421790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 650.207179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 650.704137] Lustre: server umount lustre-MDT0000 complete [ 651.751064] LustreError: 19467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.782026] LustreError: 19467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 655.711032] LustreError: 19454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771400110 with bad export cookie 9416951421212745990 [ 655.722525] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 655.729456] LustreError: 19454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 656.817784] Lustre: server umount lustre-MDT0001 complete [ 669.396702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.484104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 670.510369] Lustre: Skipped 2 previous similar messages [ 675.917936] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 686.053200] LustreError: 22470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 686.076926] LustreError: 22470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 687.523155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.166318] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 696.227311] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 700.308199] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 702.311166] Lustre: Failing over lustre-MDT0000 [ 702.433782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.437723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.444567] Lustre: Skipped 2 previous similar messages [ 702.456268] Lustre: Skipped 6 previous similar messages [ 702.637432] Lustre: server umount lustre-MDT0000 complete [ 721.677601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.761106] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.158799] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 727.525587] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 727.534835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 727.545292] Lustre: Skipped 1 previous similar message [ 727.577302] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 727.992409] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 736.262464] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 742.883447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 746.430639] Lustre: server umount lustre-MDT0000 complete [ 749.841833] LustreError: 24793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771400204 with bad export cookie 9416951421212746928 [ 749.846727] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 749.851072] LustreError: 24793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 750.609909] Lustre: server umount lustre-MDT0001 complete [ 758.402487] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 02:36:52 (1771400212) [ 772.349335] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 781.193413] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 824.960992] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 836.693217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.000487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 837.031968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 837.105587] Lustre: lustre-MDT0000: new disk, initializing [ 837.170563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 837.188512] Lustre: Skipped 2 previous similar messages [ 837.235754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 841.930681] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 851.942310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.063736] Lustre: 28454:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 852.348971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 856.780473] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 863.922580] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 868.197676] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 875.649572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 877.953541] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 878.005733] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 878.155972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 878.193739] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 882.245058] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 889.031523] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 895.503946] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 913.888820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 913.898660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.902553] LustreError: Skipped 1 previous similar message [ 913.905392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 913.905399] Lustre: Skipped 3 previous similar messages [ 913.951840] Lustre: Skipped 3 previous similar messages [ 917.853571] Lustre: server umount lustre-OST0000 complete [ 927.953822] Lustre: server umount lustre-MDT0000 complete [ 929.254463] LustreError: 28465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.306625] LustreError: 28465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 932.225440] LustreError: 28446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771400387 with bad export cookie 9416951421212747334 [ 932.228090] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 932.236037] LustreError: 28446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 932.828716] Lustre: server umount lustre-MDT0001 complete [ 943.729282] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 946.562072] Key type lgssc unregistered [ 946.819155] LNet: 32044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.823645] LNetError: 32044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.879975] LNet: Removed LNI 192.168.203.114@tcp [ 948.869149] Key type .llcrypt unregistered [ 948.871763] Key type ._llcrypt unregistered [ 973.545566] Key type ._llcrypt registered [ 973.547625] Key type .llcrypt registered [ 973.676724] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 974.807576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 974.937063] alg: No test for adler32 (adler32-zlib) [ 976.300868] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 976.603994] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 978.295334] Key type lgssc registered [ 979.509751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 989.815426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 989.835679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 991.777791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.363783] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1001.953863] LustreError: 33446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.977538] LustreError: 33446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1007.080041] LustreError: 33446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1007.869271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.201286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1013.559635] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1019.053143] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1022.973419] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1029.046045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1029.422576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1033.539430] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1035.991264] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1042.073767] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1056.866092] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 02:41:50 (1771400510) [ 1058.489543] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1060.306876] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1067.020864] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 02:42:01 (1771400521) [ 1077.027584] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 02:42:11 (1771400531) [ 1078.182971] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1087.797260] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1087.806291] Lustre: Skipped 2 previous similar messages [ 1089.912888] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 02:42:23 (1771400543) [ 1092.295692] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1092.307888] Lustre: Skipped 2 previous similar messages [ 1101.040508] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1101.049309] Lustre: Skipped 4 previous similar messages [ 1116.837041] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 02:42:50 (1771400570) [ 1120.507447] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1124.495699] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1133.312341] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 02:43:07 (1771400587) [ 1135.252719] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1135.260438] Lustre: Skipped 6 previous similar messages [ 1148.787674] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 02:43:22 (1771400602) [ 1164.775789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.790262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1164.790530] Lustre: Skipped 1 previous similar message [ 1164.792413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1164.819374] Lustre: Skipped 1 previous similar message [ 1169.532738] Lustre: server umount lustre-OST0000 complete [ 1175.013917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.024214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1175.034701] Lustre: Skipped 1 previous similar message [ 1175.063881] Lustre: Skipped 1 previous similar message [ 1180.131332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1180.142330] Lustre: Skipped 1 previous similar message [ 1184.224944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1184.235590] Lustre: Skipped 1 previous similar message [ 1188.319162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1188.565046] Lustre: server umount lustre-MDT0000 complete [ 1189.351284] LustreError: 33440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.379810] LustreError: 33440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1192.226592] LustreError: 33427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771400647 with bad export cookie 17968836868515091498 [ 1192.226592] LustreError: 33470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771400647 with bad export cookie 17968836868515091498 [ 1192.228936] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1192.236510] LustreError: 33427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1192.707624] Lustre: server umount lustre-MDT0001 complete [ 1204.155198] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1207.447960] Key type lgssc unregistered [ 1207.832495] LNet: 40960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.845662] LNetError: 40960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.870617] LNet: Removed LNI 192.168.203.114@tcp [ 1208.794641] Key type .llcrypt unregistered [ 1208.797856] Key type ._llcrypt unregistered [ 1227.714984] Key type ._llcrypt registered [ 1227.720376] Key type .llcrypt registered [ 1227.877748] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 1245.223511] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1246.412117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1246.446721] alg: No test for adler32 (adler32-zlib) [ 1247.541486] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 1247.821072] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1249.520111] Key type lgssc registered [ 1250.928656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1288.717196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1288.803838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.281116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1290.338726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1290.465407] Lustre: lustre-MDT0000: new disk, initializing [ 1290.637448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.693468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1295.154073] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1306.713061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1306.822208] Lustre: 44081:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1306.855791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1306.858928] Lustre: Skipped 1 previous similar message [ 1306.913570] Lustre: lustre-MDT0001: new disk, initializing [ 1306.996721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1307.043155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1307.071643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1312.904823] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1320.792115] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1324.680598] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1333.324178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1333.689914] Lustre: lustre-OST0000: new disk, initializing [ 1333.692699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1333.773650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1340.501874] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1343.565061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1343.580536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1343.744993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1348.082522] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1353.831589] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1354.101664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1359.907451] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1360.195777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1364.452497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1364.456849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1364.461180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1364.469833] Lustre: Skipped 1 previous similar message [ 1368.715558] Lustre: server umount lustre-OST0000 complete [ 1373.665248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1373.681153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1373.696413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1373.701696] Lustre: Skipped 1 previous similar message [ 1374.690104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.722864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1374.728465] Lustre: Skipped 1 previous similar message [ 1379.520918] Lustre: server umount lustre-MDT0000 complete [ 1383.969695] LustreError: 44076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771400838 with bad export cookie 8190047009918871949 [ 1383.979232] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1384.412625] Lustre: server umount lustre-MDT0001 complete [ 1396.636711] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 02:47:30 (1771400850) [ 1404.891388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1405.369417] LustreError: 48137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.378131] LustreError: 48137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1405.423738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1410.016222] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1410.528831] LustreError: 48137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.653472] LustreError: 48137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1418.615881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1418.945888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1423.639875] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1429.739688] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1434.239681] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1441.441347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1441.826357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.418934] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1455.558802] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1460.207590] LustreError: 50328:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1461.003627] Lustre: Modifying parameter general.timeout=1 in log params [ 1462.654433] Lustre: Modifying parameter general.timeout=3 in log params [ 1462.659894] Lustre: Skipped 1 previous similar message [ 1465.324568] Lustre: Modifying parameter general.timeout=6 in log params [ 1465.331097] Lustre: Skipped 2 previous similar messages [ 1470.076948] Lustre: Modifying parameter general.timeout=12 in log params [ 1470.082115] Lustre: Skipped 5 previous similar messages [ 1478.500888] Lustre: Modifying parameter general.timeout=22 in log params [ 1478.503309] Lustre: Skipped 9 previous similar messages [ 1494.872222] Lustre: Modifying parameter general.timeout=40 in log params [ 1494.879901] Lustre: Skipped 17 previous similar messages [ 1527.556356] Lustre: Modifying parameter general.timeout=74 in log params [ 1527.560930] Lustre: Skipped 33 previous similar messages [ 1594.600894] Lustre: Disabling parameter general.timeout= in log params [ 1594.603772] Lustre: Skipped 55 previous similar messages [ 1598.428694] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 02:50:51 (1771401051) [ 1601.842171] Lustre: 57276:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1603.889163] Lustre: 57383:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1606.089986] Lustre: 57486:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1612.170577] Lustre: 57781:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1612.185451] Lustre: 57781:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 1643.320638] Lustre: 59153:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1643.387996] LustreError: 59153:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1648.240481] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 02:51:42 (1771401102) [ 1664.996836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1665.010939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1665.031475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1671.142754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1671.145511] Lustre: Skipped 3 previous similar messages [ 1671.518996] Lustre: server umount lustre-MDT0000 complete [ 1675.520805] LustreError: 48116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771401130 with bad export cookie 8190047009918872831 [ 1675.531400] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1675.532677] LustreError: 48116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1675.802636] Lustre: server umount lustre-MDT0001 complete [ 1690.133294] Lustre: server umount lustre-OST0000 complete [ 1718.338403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.561516] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1718.908540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.125144] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1733.630454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.766112] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1733.808453] Lustre: 61686:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1738.743183] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1743.962796] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1747.709367] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1755.312455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1755.551317] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1755.837557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.850092] Lustre: Skipped 1 previous similar message [ 1762.978746] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1769.337710] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1770.902169] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1770.917164] Lustre: Skipped 23 previous similar messages [ 1770.999957] Lustre: 63913:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1782.248422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1782.253029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.267027] LustreError: Skipped 1 previous similar message [ 1782.269941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.302702] Lustre: Skipped 3 previous similar messages [ 1785.690260] Lustre: server umount lustre-OST0000 complete [ 1790.432107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1790.442345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.030316] Lustre: server umount lustre-MDT0000 complete [ 1797.605750] LustreError: 61700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1797.632682] LustreError: 61700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1800.526358] LustreError: 61681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771401255 with bad export cookie 8190047009918875939 [ 1800.528461] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1800.539730] LustreError: 61681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1800.924891] Lustre: server umount lustre-MDT0001 complete [ 1811.337530] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1814.229857] Key type lgssc unregistered [ 1814.532513] LNet: 65104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.543737] LNetError: 65104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.567821] LNet: Removed LNI 192.168.203.114@tcp [ 1815.388146] Key type .llcrypt unregistered [ 1815.391340] Key type ._llcrypt unregistered [ 1839.849252] Key type ._llcrypt registered [ 1839.852787] Key type .llcrypt registered [ 1839.956757] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1841.056456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1841.155138] alg: No test for adler32 (adler32-zlib) [ 1842.201875] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 1842.377817] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1844.047159] Key type lgssc registered [ 1845.079629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1853.373653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1853.401109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.108330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.032982] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1864.673638] LustreError: 66500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1864.697053] LustreError: 66500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1869.804447] LustreError: 66500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1871.476704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1872.547023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1878.494196] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1883.770674] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1887.519328] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1894.568799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1895.050593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.377192] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1906.947466] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1913.726093] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1926.114836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.131387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1926.140706] Lustre: Skipped 2 previous similar messages [ 1926.155843] Lustre: Skipped 2 previous similar messages [ 1930.250827] Lustre: server umount lustre-MDT0000 complete [ 1931.245618] LustreError: 67214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1931.282081] LustreError: 67214:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1933.626288] LustreError: 66479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771401388 with bad export cookie 11257543113127947309 [ 1933.633871] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1933.638600] LustreError: 66479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1933.969051] Lustre: server umount lustre-MDT0001 complete [ 1948.077602] Lustre: server umount lustre-OST0000 complete [ 1974.072566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1974.367586] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1974.387236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1974.628325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.607419] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1987.686446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.763734] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1987.782812] Lustre: 71039:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1988.175131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1992.564060] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1997.493666] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2001.263237] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2008.706858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2008.935063] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2009.307564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2016.545130] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2021.581443] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2023.360287] Lustre: 73274:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2025.218845] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2031.590463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2031.601249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.613742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2036.193701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2036.204185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.225741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2037.609532] Lustre: server umount lustre-OST0000 complete [ 2041.832151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.848037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2041.855402] Lustre: Skipped 2 previous similar messages [ 2046.947194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2046.958450] Lustre: Skipped 1 previous similar message [ 2047.621441] Lustre: server umount lustre-MDT0000 complete [ 2050.994904] LustreError: 71030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771401505 with bad export cookie 11257543113127948156 [ 2051.003779] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2051.289989] Lustre: server umount lustre-MDT0001 complete [ 2062.676755] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 2065.468646] Key type lgssc unregistered [ 2065.739860] LNet: 74466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.749342] LNetError: 74466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2065.773083] LNet: Removed LNI 192.168.203.114@tcp [ 2066.509550] Key type .llcrypt unregistered [ 2066.512479] Key type ._llcrypt unregistered [ 2088.309304] Key type ._llcrypt registered [ 2088.311661] Key type .llcrypt registered [ 2088.420765] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2089.842835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2089.871173] alg: No test for adler32 (adler32-zlib) [ 2090.957860] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 2091.151958] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 2092.823160] Key type lgssc registered [ 2093.749634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2100.829406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2100.855295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2102.313490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2105.986825] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2112.499955] LustreError: 75861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2112.517604] LustreError: 75861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2114.630887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.067358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2119.215664] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2124.529530] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2128.121201] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2134.781452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2135.372219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.872639] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2147.751647] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2152.893572] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2153.783531] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2155.581563] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2155.587723] Lustre: Skipped 1 previous similar message [ 2158.242689] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2158.251593] Lustre: Skipped 2 previous similar messages [ 2162.560724] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2162.567297] Lustre: Skipped 4 previous similar messages [ 2170.795373] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2170.800780] Lustre: Skipped 9 previous similar messages [ 2187.066299] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2187.069817] Lustre: Skipped 18 previous similar messages [ 2219.196414] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2219.201467] Lustre: Skipped 36 previous similar messages [ 2247.272831] Lustre: 83634:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2249.653389] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 03:01:43 (1771401703) [ 2251.123541] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2252.593956] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 03:01:47 (1771401707) [ 2257.376112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2257.385188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.398824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2260.452815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2260.471056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2260.478348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2261.744156] Lustre: server umount lustre-OST0000 complete [ 2265.576859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.591608] Lustre: Skipped 1 previous similar message [ 2265.595509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2265.603681] Lustre: Skipped 2 previous similar messages [ 2270.688997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2270.699158] Lustre: Skipped 1 previous similar message [ 2275.809975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2275.825842] Lustre: Skipped 1 previous similar message [ 2279.391123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2279.574049] Lustre: server umount lustre-MDT0000 complete [ 2280.941286] LustreError: 75862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2280.954033] LustreError: 75862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2282.705903] LustreError: 75843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771401737 with bad export cookie 6586393424814706683 [ 2282.717537] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2283.118249] Lustre: server umount lustre-MDT0001 complete [ 2293.290371] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 2295.771895] Key type lgssc unregistered [ 2296.015411] LNet: 84867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.022767] LNetError: 84867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2297.061830] LNet: Removed LNI 192.168.203.114@tcp [ 2298.010471] Key type .llcrypt unregistered [ 2298.014922] Key type ._llcrypt unregistered [ 2302.656428] Key type ._llcrypt registered [ 2302.659701] Key type .llcrypt registered [ 2302.775312] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2307.083000] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules [ 2308.611335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2308.654206] alg: No test for adler32 (adler32-zlib) [ 2309.742598] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 2310.034765] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 2311.703413] Key type lgssc registered [ 2313.176675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2314.000493] LustreError: 86036:0:(lu_object.c:1434:lu_context_key_register()) cfs_race id 60d sleeping [ 2329.442295] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2335.602406] LustreError: 86036:0:(lu_object.c:1434:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2337.140124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2337.151360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2338.594305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2342.613805] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2349.027322] LustreError: 87042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.046287] LustreError: 87042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2350.204709] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 03:03:24 (1771401804) [ 2354.017436] LustreError: 87074:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2354.055120] LustreError: 87074:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2354.063664] LustreError: 87074:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2354.272734] Lustre: server umount lustre-MDT0000 complete [ 2364.044953] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 2365.948042] Key type lgssc unregistered [ 2366.194800] LNet: 88535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2366.209306] LNetError: 88535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2367.269941] LNet: Removed LNI 192.168.203.114@tcp [ 2367.925886] Key type .llcrypt unregistered [ 2367.928153] Key type ._llcrypt unregistered [ 2382.780786] Key type ._llcrypt registered [ 2382.782952] Key type .llcrypt registered [ 2382.901845] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2383.776372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2383.839747] alg: No test for adler32 (adler32-zlib) [ 2384.852627] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 2385.091121] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 2386.759242] Key type lgssc registered [ 2387.781638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2394.247053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2394.268135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.748473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2399.807413] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2405.856655] LustreError: 89735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.874197] LustreError: 89735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2407.726941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.315292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2412.349401] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2417.607440] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2422.065988] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2428.338695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2429.026433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.547474] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2439.551460] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2486.697416] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 03:05:40 (1771401940) [ 2488.485724] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2490.795444] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 03:05:44 (1771401944) [ 2495.456471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2495.459899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2495.491659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2496.995515] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2497.001732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2497.013140] Lustre: Skipped 1 previous similar message [ 2497.029585] Lustre: Skipped 2 previous similar messages [ 2500.065473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2500.081081] Lustre: Skipped 2 previous similar messages [ 2501.490886] Lustre: server umount lustre-MDT0000 complete [ 2504.648717] LustreError: 90457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771401959 with bad export cookie 10421698801352518834 [ 2504.655151] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2504.655992] LustreError: 90457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2504.964302] Lustre: server umount lustre-MDT0001 complete [ 2519.043065] Lustre: server umount lustre-OST0000 complete [ 2530.549509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.901601] LustreError: 93673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.060101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.652445] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2536.426466] LustreError: 93672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.454848] LustreError: 93672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2540.515160] LustreError: 93672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2541.374365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2541.868850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2544.787500] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2549.401253] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2552.916919] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2563.192604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2563.352756] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2563.357492] LustreError: 93660:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2563.370505] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2563.375233] LustreError: 95528:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 2563.379336] LustreError: 95528:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2563.383260] LustreError: 95528:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2563.429157] Lustre: server umount lustre-OST0000 complete [ 2563.432152] LustreError: 95528:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2570.990625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2571.180483] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2571.184846] LustreError: 93661:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2571.190746] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2571.197930] LustreError: 95989:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 2571.204719] LustreError: 95989:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2571.209825] LustreError: 95989:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2571.261954] Lustre: server umount lustre-OST0000 complete [ 2571.264870] LustreError: 95989:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2579.196237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2579.315943] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2579.319501] Lustre: Found index 0 for lustre-OST0000, updating log [ 2579.323763] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2579.376914] Lustre: lustre-OST0000: new disk, initializing [ 2579.387906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2579.674855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.100586] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2588.360745] Lustre: Failing over lustre-OST0000 [ 2588.473783] Lustre: server umount lustre-OST0000 complete [ 2592.741525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2592.750551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.767901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.347509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.356402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.380571] Lustre: Skipped 1 previous similar message [ 2598.783612] Lustre: server umount lustre-MDT0000 complete [ 2601.774169] LustreError: 93652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771402056 with bad export cookie 10421698801352521578 [ 2601.785548] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2601.786331] LustreError: 93652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2611.321427] Lustre: server umount lustre-MDT0001 complete [ 2619.175304] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 03:07:53 (1771402073) [ 2633.657695] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2642.602409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.947299] LustreError: 98838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.021639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2646.553217] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2652.141470] LustreError: 98837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.164698] LustreError: 98837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2653.607657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.242693] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2660.824216] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2666.955629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2667.432960] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2671.805924] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2672.620029] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2672.626050] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2672.635279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2673.189169] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2677.737317] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2677.748909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -16 [ 2677.753204] LustreError: Skipped 1 previous similar message [ 2678.602045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2678.688550] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2678.779640] Lustre: lustre-OST0001: new disk, initializing [ 2678.786205] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2678.788441] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2678.788701] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2678.995130] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2679.000831] Lustre: Skipped 2 previous similar messages [ 2681.580647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2681.597840] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2681.808969] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2684.162975] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2690.425403] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2695.701160] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2705.888372] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2705.892615] LustreError: Skipped 1 previous similar message [ 2705.894064] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.905886] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2710.195992] Lustre: server umount lustre-MDT0001 complete [ 2711.802232] LustreError: 101572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.824757] LustreError: 101572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2717.862883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2718.041252] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2718.044467] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2718.094943] Lustre: 98827:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2719.733921] LustreError: 102675:0:(obd_mount.c:212:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2719.742124] LustreError: 102675:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2719.755518] LustreError: 98830:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2719.856435] LustreError: 102681:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.203.114@tcp: cfg command failed: rc = -17 [ 2719.861816] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2719.861816] [ 2722.000421] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2723.846447] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2731.973926] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 03:09:45 (1771402185) [ 2746.911896] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2761.497283] Lustre: 104676:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2780.339749] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2783.708382] Lustre: 105810:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2862.057050] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.058995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2862.067743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2862.067752] Lustre: Skipped 4 previous similar messages [ 2862.079906] Lustre: Skipped 3 previous similar messages [ 2867.850906] Lustre: server umount lustre-MDT0000 complete [ 2870.233494] LustreError: 102584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771402325 with bad export cookie 10421698801352522509 [ 2870.235463] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2870.237970] LustreError: 102584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2870.523682] Lustre: server umount lustre-MDT0001 complete [ 2883.712977] Lustre: server umount lustre-OST0000 complete [ 2896.840773] Lustre: server umount lustre-OST0001 complete [ 2900.311186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2906.259203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2911.597840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2918.153839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2931.137996] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2938.713344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2938.730691] Lustre: lustre-MDT0000: reset Object Index mappings [ 2939.042437] LustreError: 109747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2939.052852] LustreError: 109747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2939.123775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.129352] Lustre: Skipped 1 previous similar message [ 2942.460555] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2948.297122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.309991] Lustre: lustre-MDT0001: reset Object Index mappings [ 2951.623449] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2953.776635] Lustre: 110857:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2958.594050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2960.655535] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:227 to 0x280000400:257) [ 2962.907404] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2965.489363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2968.058668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2970.256155] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 2970.256930] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2971.811801] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2977.553224] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2980.010861] Lustre: 112707:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3018.282439] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 03:14:32 (1771402472) [ 3050.474428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3050.478599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3050.481030] Lustre: Skipped 5 previous similar messages [ 3050.486955] Lustre: Skipped 10 previous similar messages [ 3052.582328] Lustre: server umount lustre-MDT0000 complete [ 3055.325226] LustreError: 109728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771402510 with bad export cookie 10421698801352569339 [ 3055.330278] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3055.334633] LustreError: 109728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3055.584754] LustreError: 109742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3055.601096] LustreError: 109742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3061.667166] Lustre: server umount lustre-MDT0001 complete [ 3070.442858] Lustre: server umount lustre-OST0000 complete [ 3087.327408] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3087.464918] Lustre: server umount lustre-OST0001 complete [ 3092.319590] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 3100.065587] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3120.616540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.750425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3120.766363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3120.807873] Lustre: lustre-MDT0000: new disk, initializing [ 3120.864327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3120.870476] Lustre: Skipped 3 previous similar messages [ 3120.886668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3122.997224] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3127.539521] Lustre: server umount lustre-MDT0000 complete [ 3132.418550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.617140] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3132.624583] Lustre: 116919:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3132.632685] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3134.923095] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3137.703405] Lustre: server umount lustre-MDT0000 complete [ 3143.439651] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 03:16:38 (1771402598) [ 3144.396093] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3145.636254] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 03:16:40 (1771402600) [ 3155.991554] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3161.329544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.148814] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3169.018515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.077740] Lustre: 118932:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3169.117845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3169.121776] Lustre: Skipped 1 previous similar message [ 3169.163553] Lustre: lustre-MDT0001: new disk, initializing [ 3169.243945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3169.247613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3171.215377] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3174.255400] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3177.844253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3177.991840] Lustre: lustre-OST0000: new disk, initializing [ 3177.996321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3179.491269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3179.500535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3179.581683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3181.842766] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3188.777572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3188.867279] Lustre: lustre-OST0001: new disk, initializing [ 3188.870295] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3189.997332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3190.008493] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3190.059908] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3192.043874] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3198.401788] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3200.408183] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3420.787043] Lustre: 127696:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3420.792282] Lustre: 127696:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3420.797456] Lustre: 127696:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3420.804222] Lustre: 127696:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3420.808687] Lustre: 127696:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3420.812807] Lustre: 127696:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3566.500819] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:23:41 (1771403021) [ 3578.848734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3578.852020] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3578.856949] Lustre: Skipped 2 previous similar messages [ 3578.859690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3578.861769] Lustre: Skipped 4 previous similar messages [ 3580.501119] Lustre: server umount lustre-MDT0000 complete [ 3582.172807] LustreError: 118925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771403037 with bad export cookie 10421698801352631996 [ 3582.180201] LustreError: 118925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3582.180221] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3584.481183] LustreError: 122117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3584.493240] LustreError: 122117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3588.380087] Lustre: server umount lustre-MDT0001 complete [ 3596.103174] Lustre: server umount lustre-OST0000 complete [ 3612.127130] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3612.197541] Lustre: server umount lustre-OST0001 complete [ 3614.972390] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 3619.012327] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3633.659839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.762888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3633.775351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3633.811689] Lustre: lustre-MDT0000: new disk, initializing [ 3633.847236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.850151] Lustre: Skipped 5 previous similar messages [ 3633.857740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3635.500372] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3640.408242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.464431] Lustre: 130687:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3640.583024] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3642.317514] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3646.032920] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3647.652572] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3650.352834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3651.769843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3651.773775] Lustre: Skipped 1 previous similar message [ 3651.775820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3651.811202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3652.714366] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3656.178787] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3659.813337] Lustre: lustre-MDD0000: changelog on [ 3660.963813] Lustre: lustre-MDD0001: changelog on [ 4286.986097] Lustre: lustre-MDD0001: changelog off [ 4287.615138] Lustre: 134768:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4287.916169] Lustre: lustre-MDD0000: changelog off [ 4292.066968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4292.067060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4292.067772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4292.067777] Lustre: Skipped 8 previous similar messages [ 4292.070431] LustreError: Skipped 1 previous similar message [ 4292.073679] Lustre: Skipped 6 previous similar messages [ 4295.517936] Lustre: server umount lustre-OST0000 complete [ 4303.175749] Lustre: server umount lustre-MDT0000 complete [ 4304.431269] LustreError: 130679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771403759 with bad export cookie 10421698801353443996 [ 4304.434605] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4304.435096] LustreError: 130679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4304.546457] Lustre: server umount lustre-MDT0001 complete [ 4308.062996] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 4309.141692] Key type lgssc unregistered [ 4309.291156] LNet: 141041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4309.294248] LNetError: 141041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4309.305376] LNet: Removed LNI 192.168.203.114@tcp [ 4309.625129] Key type .llcrypt unregistered [ 4309.626152] Key type ._llcrypt unregistered [ 4316.555384] Key type ._llcrypt registered [ 4316.556657] Key type .llcrypt registered [ 4316.599748] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 4323.069295] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4323.412386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4323.419781] alg: No test for adler32 (adler32-zlib) [ 4324.286882] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 4324.377731] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 4325.975239] Key type lgssc registered [ 4326.396489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4339.471177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4339.477435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4340.566664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4340.579937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4340.617134] Lustre: lustre-MDT0000: new disk, initializing [ 4340.652217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4340.659134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4341.984108] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4346.672151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4346.714237] Lustre: 144196:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4346.737600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4346.739603] Lustre: Skipped 1 previous similar message [ 4346.772679] Lustre: lustre-MDT0001: new disk, initializing [ 4346.810966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4346.823585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4346.827757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4348.205786] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4351.394963] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4352.589087] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4354.823060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4354.945263] Lustre: lustre-OST0000: new disk, initializing [ 4354.947636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4354.985618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4355.301094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4355.304820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4355.345369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4357.182155] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4360.369225] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4362.585606] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_module obdecho/obdecho [ 4363.216688] LustreError: 146810:0:(echo_client.c:670:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4363.220358] LustreError: 146810:0:(obd_config.c:880:class_setup()) setup ec failed (-22) [ 4363.826935] LustreError: 146907:0:(echo_client.c:1879:echo_md_handler()) Can not resolve the path : rc = -2 [ 4367.327782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4367.330608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4367.335821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4370.913202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4370.914055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4370.915563] Lustre: Skipped 1 previous similar message [ 4370.921392] Lustre: Skipped 1 previous similar message [ 4373.217571] Lustre: server umount lustre-MDT0000 complete [ 4374.630519] LustreError: 144189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771403829 with bad export cookie 2937426637972753704 [ 4374.633770] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4374.638372] LustreError: 144189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4374.753599] Lustre: server umount lustre-MDT0001 complete [ 4386.458274] Lustre: server umount lustre-OST0000 complete [ 4393.832467] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 4397.550610] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4410.995302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4411.088641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4411.102226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4411.143365] Lustre: lustre-MDT0000: new disk, initializing [ 4411.176258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4411.188538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4412.668624] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4417.335661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4417.376767] Lustre: 150579:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4417.401388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4417.403582] Lustre: Skipped 1 previous similar message [ 4417.439151] Lustre: lustre-MDT0001: new disk, initializing [ 4417.480382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4417.492184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4417.496363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4418.878584] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4422.244621] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4423.502752] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4426.001753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4426.103047] Lustre: lustre-OST0000: new disk, initializing [ 4426.105362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4426.136748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.246399] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4431.399740] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4432.801733] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4435.959419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4435.963349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4436.000842] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4436.971340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4438.507112] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4438.591700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4441.056479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4441.062243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4441.068980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4441.071525] Lustre: Skipped 2 previous similar messages [ 4445.535788] Lustre: server umount lustre-OST0000 complete [ 4448.223894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4448.227038] LustreError: Skipped 1 previous similar message [ 4448.228910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.233071] Lustre: Skipped 1 previous similar message [ 4448.234968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4453.087760] Lustre: server umount lustre-MDT0000 complete [ 4454.515131] LustreError: 150571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771403909 with bad export cookie 2937426637972754649 [ 4454.516280] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4454.521367] LustreError: 150571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4454.665491] Lustre: server umount lustre-MDT0001 complete [ 4456.153585] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 03:38:31 (1771403911) [ 4456.853229] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4457.604043] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 03:38:32 (1771403912) [ 4460.593490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4460.802216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4462.245959] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4465.840781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4467.556205] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4469.589705] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4471.029140] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4473.881659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4476.070394] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4478.049573] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4490.721389] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4490.724746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4490.727306] Lustre: Skipped 3 previous similar messages [ 4490.729580] Lustre: Skipped 3 previous similar messages [ 4496.738829] Lustre: server umount lustre-MDT0000 complete [ 4498.252906] LustreError: 156225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771403953 with bad export cookie 2937426637972755517 [ 4498.255382] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4498.257785] LustreError: 156225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4498.673484] LustreError: 154605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4498.680032] LustreError: 154605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4500.959928] LustreError: 154605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.960299] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4500.971368] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4500.975404] Lustre: Skipped 5 previous similar messages [ 4503.793663] LustreError: 154605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4504.433865] Lustre: server umount lustre-MDT0001 complete [ 4506.870303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4511.298243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4511.385776] LustreError: 158132:0:(llog.c:1643:llog_backup()) MGC192.168.203.114@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4511.389312] Lustre: 158132:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.114@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4528.369395] LustreError: 158138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.376274] LustreError: 158138:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4533.490169] LustreError: 158138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4533.496814] LustreError: 158138:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4543.968028] LustreError: 158138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4543.973044] LustreError: 158138:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4560.117722] LustreError: 158138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4560.124979] LustreError: 158138:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 4572.255279] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4572.257410] Lustre: 158132:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4593.120400] LustreError: 158158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4593.123672] LustreError: 158158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 4600.287277] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4600.289130] Lustre: 154603:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4613.094270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4613.097119] Lustre: Skipped 2 previous similar messages [ 4614.474873] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4617.385239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4618.211360] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4618.213957] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x28c3d5f6ce22ed73 [ 4619.735638] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4621.580487] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4622.149049] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4623.333403] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4623.336107] Lustre: Skipped 1 previous similar message [ 4623.459097] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4627.083818] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4633.568881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4633.569504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4633.570547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4633.570552] Lustre: Skipped 2 previous similar messages [ 4633.579161] Lustre: Skipped 2 previous similar messages [ 4633.944315] Lustre: server umount lustre-MDT0000 complete [ 4635.265105] LustreError: 154593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771404090 with bad export cookie 2937426637972761971 [ 4635.267676] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4635.268961] LustreError: 154593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4635.385454] Lustre: server umount lustre-MDT0001 complete [ 4646.951342] Lustre: server umount lustre-OST0000 complete [ 4654.247988] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 4657.513422] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4670.259798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4670.354234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4670.367522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4670.403155] Lustre: lustre-MDT0000: new disk, initializing [ 4670.435259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4671.838865] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4676.335557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4676.381796] Lustre: 162948:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4676.455676] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4676.461948] Lustre: Skipped 2 previous similar messages [ 4676.504612] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4678.047997] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4681.809638] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4683.382315] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4686.107034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4686.285836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4686.288620] Lustre: Skipped 3 previous similar messages [ 4688.593389] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4691.842448] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4692.473813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4692.514181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4693.238393] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4693.305238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4694.840202] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4694.924970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4697.568649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4697.570303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.570764] LustreError: Skipped 1 previous similar message [ 4701.718957] Lustre: server umount lustre-OST0000 complete [ 4707.295847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4709.083459] Lustre: server umount lustre-MDT0000 complete [ 4710.386560] LustreError: 164686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771404165 with bad export cookie 2937426637972778771 [ 4710.390832] LustreError: 164686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4710.391220] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4710.509919] Lustre: server umount lustre-MDT0001 complete [ 4713.474479] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:42:48 (1771404168) [ 4715.866577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4716.036341] LustreError: 166952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4716.045873] LustreError: 166952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 4717.455385] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4720.337361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.822293] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4723.614438] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4724.908184] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4727.201261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4729.177164] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4730.857275] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4750.282639] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:43:25 (1771404205) [ 4751.842780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4751.843134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4751.847303] Lustre: Skipped 5 previous similar messages [ 4757.329781] Lustre: server umount lustre-OST0000 complete [ 4762.080895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4762.086092] Lustre: Skipped 6 previous similar messages [ 4773.343132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4773.398775] Lustre: server umount lustre-MDT0000 complete [ 4774.608243] LustreError: 167668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771404229 with bad export cookie 2937426637972779653 [ 4774.611569] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4774.612602] LustreError: 167668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4774.731935] Lustre: server umount lustre-MDT0001 complete [ 4778.237523] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 4779.225932] Key type lgssc unregistered [ 4779.346352] LNet: 170348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4779.349225] LNetError: 170348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4779.356307] LNet: Removed LNI 192.168.203.114@tcp [ 4779.643724] Key type .llcrypt unregistered [ 4779.645034] Key type ._llcrypt unregistered [ 4786.741953] Key type ._llcrypt registered [ 4786.743415] Key type .llcrypt registered [ 4786.784113] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4787.091077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4787.123584] alg: No test for adler32 (adler32-zlib) [ 4787.971328] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 4788.048580] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 4789.623145] Key type lgssc registered [ 4789.988950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4792.665178] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4792.671254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4808.095250] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4813.215205] LustreError: 171550:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.114@tcp: failed processing log, type 4: rc = -110 [ 4818.335369] LustreError: 171550:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4818.340379] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4818.343061] LustreError: MGC192.168.203.114@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4818.346485] LustreError: 171550:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4818.349180] LustreError: 171550:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -22 [ 4818.351538] LustreError: 171550:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4818.374348] Lustre: server umount lustre-OST0000 complete [ 4818.375585] LustreError: 171550:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4821.267256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4822.412317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4823.612262] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4826.516436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4826.648379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4827.876233] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4829.805535] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4831.112286] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4833.462500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4833.545287] LustreError: 173636:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4833.550426] Lustre: 173636:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.203.114@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4833.600876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4835.556201] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4837.406160] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4841.956546] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4841.957839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4841.962106] Lustre: Skipped 2 previous similar messages [ 4841.963951] Lustre: Skipped 1 previous similar message [ 4841.966645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4844.306874] Lustre: server umount lustre-MDT0000 complete [ 4845.503180] LustreError: 172029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771404300 with bad export cookie 245594960031601197 [ 4845.505459] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4845.507431] LustreError: 172029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4845.608860] Lustre: server umount lustre-MDT0001 complete [ 4856.219495] Lustre: server umount lustre-OST0000 complete [ 4859.196292] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 4862.198422] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4874.393777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4874.492366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4874.502336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4874.538995] Lustre: lustre-MDT0000: new disk, initializing [ 4874.564142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4874.572065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4875.874523] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4879.931575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4879.967759] Lustre: 177010:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4879.991416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4879.993093] Lustre: Skipped 1 previous similar message [ 4880.026931] Lustre: lustre-MDT0001: new disk, initializing [ 4880.059038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4880.070927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4880.074542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4881.298697] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4884.320940] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4885.494395] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4887.625957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4887.718306] Lustre: lustre-OST0000: new disk, initializing [ 4887.720134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4888.821765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4888.825038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4888.859838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4889.598655] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4892.697029] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4894.065014] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4894.134702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4895.481607] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4895.549226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4898.253849] Lustre: server umount lustre-OST0000 complete [ 4900.831904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4900.834349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4900.838228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4900.840316] Lustre: Skipped 1 previous similar message [ 4904.416168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4904.416788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4909.536199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4909.539232] Lustre: Skipped 2 previous similar messages [ 4909.561336] Lustre: server umount lustre-MDT0000 complete [ 4910.762028] LustreError: 177001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771404365 with bad export cookie 245594960031601883 [ 4910.765800] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4920.336668] Lustre: server umount lustre-MDT0001 complete [ 4923.226262] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:46:18 (1771404378) [ 4929.102918] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 4930.097737] Key type lgssc unregistered [ 4930.221334] LNet: 181528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4930.223635] LNetError: 181528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4930.231569] LNet: Removed LNI 192.168.203.114@tcp [ 4930.510897] Key type .llcrypt unregistered [ 4930.512388] Key type ._llcrypt unregistered [ 4937.621233] Key type ._llcrypt registered [ 4937.622136] Key type .llcrypt registered [ 4937.672888] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4938.118933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4938.197371] alg: No test for adler32 (adler32-zlib) [ 4939.064517] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 4939.142415] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 4940.719112] Key type lgssc registered [ 4941.094251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4943.683019] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4943.688507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4944.838861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4946.089739] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4948.945280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4949.066197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4950.272660] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4951.943885] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4953.219118] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4955.812632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4955.888534] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4955.889910] LustreError: 184339:0:(llog.c:1684:llog_backup()) MGC192.168.203.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4955.892432] Lustre: 184339:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.203.114@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4955.932352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4957.766504] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4959.483969] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4964.319836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4964.322419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4964.327043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4965.344802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4965.345940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4965.347210] Lustre: Skipped 1 previous similar message [ 4965.352475] Lustre: Skipped 1 previous similar message [ 4966.355911] Lustre: server umount lustre-MDT0000 complete [ 4967.541554] LustreError: 182685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771404422 with bad export cookie 10190442947886103851 [ 4967.544685] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4967.545981] LustreError: 182685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4967.650509] Lustre: server umount lustre-MDT0001 complete [ 4979.093140] Lustre: server umount lustre-OST0000 complete [ 4981.937291] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 4984.910594] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4996.028629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4996.120557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4996.134446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4996.170052] Lustre: lustre-MDT0000: new disk, initializing [ 4996.194203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.200336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4997.405762] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5001.345760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5001.386800] Lustre: 187722:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5001.404554] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5001.405706] Lustre: Skipped 1 previous similar message [ 5001.406782] LustreError: 188486:0:(llog.c:1684:llog_backup()) MGC192.168.203.114@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5001.409398] LustreError: 188486:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 5001.411364] Lustre: 188486:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.114@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5001.418200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5001.419793] Lustre: Skipped 1 previous similar message [ 5001.453630] Lustre: lustre-MDT0001: new disk, initializing [ 5001.480960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5001.495605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5001.498995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5002.773946] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5005.669509] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5006.768498] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5008.879680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5008.963144] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5008.964491] Lustre: Skipped 1 previous similar message [ 5008.965496] LustreError: 189438:0:(llog.c:1684:llog_backup()) MGC192.168.203.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5008.967741] LustreError: 189438:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 5008.971180] Lustre: 189438:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.114@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5008.973646] Lustre: 189438:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5008.981290] Lustre: lustre-OST0000: new disk, initializing [ 5008.982906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5010.720322] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5013.658113] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5014.960960] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5016.062142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5016.067364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5016.095943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5017.084853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5018.441719] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5018.510330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5021.153573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5021.153970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.156244] LustreError: Skipped 1 previous similar message [ 5021.162381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5021.166094] Lustre: Skipped 1 previous similar message [ 5025.230329] Lustre: server umount lustre-OST0000 complete [ 5027.296214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5027.300068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5027.304042] Lustre: Skipped 1 previous similar message [ 5027.306094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5031.392059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5031.392625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5032.736661] Lustre: server umount lustre-MDT0000 complete [ 5034.019381] LustreError: 188500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771404488 with bad export cookie 10190442947886104537 [ 5034.022958] LustreError: 188500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5034.023807] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5034.144300] Lustre: server umount lustre-MDT0001 complete [ 5037.060221] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:48:12 (1771404492) [ 5042.452142] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 5045.607166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5045.727313] LustreError: 192437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5045.734130] LustreError: 192437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5045.740851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5045.742685] Lustre: Skipped 1 previous similar message [ 5046.974104] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5049.723683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5051.001124] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5051.934743] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5054.173425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5056.088528] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5059.046580] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5059.133320] Lustre: lustre-OST0001: new disk, initializing [ 5059.135904] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5060.265390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5060.269872] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5060.288134] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5061.077499] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5066.443738] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5069.965285] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5075.648716] Lustre: DEBUG MARKER: ADD OST3 [ 5077.317958] loop: module loaded [ 5077.337608] loop0: detected capacity change from 0 to 204800000 [ 5077.424920] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5077.429243] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5077.433877] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5079.986086] loop0: detected capacity change from 0 to 204800000 [ 5082.419994] Lustre: DEBUG MARKER: STOP OST3 [ 5082.630110] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5082.697517] Lustre: lustre-OST0003: new disk, initializing [ 5082.701741] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5082.746571] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5082.749240] Lustre: Skipped 3 previous similar messages [ 5084.311516] Lustre: *** cfs_fail_loc=2109, val=2*** [ 5084.349081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5084.352221] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5084.363636] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 5084.581931] Lustre: Failing over lustre-OST0003 [ 5084.607853] LustreError: 194884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5084.632714] Lustre: server umount lustre-OST0003 complete [ 5084.767322] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5086.367077] LustreError: 196911:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 5089.174278] Lustre: DEBUG MARKER: START OST3 again [ 5089.766719] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.767717] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5089.773779] LustreError: 193888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5089.782037] LustreError: 193888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5091.550649] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5091.640854] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5093.503978] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5093.547081] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5093.554123] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5093.554621] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5106.708668] Lustre: Failing over lustre-OST0003 [ 5106.741178] Lustre: server umount lustre-OST0003 complete [ 5107.173855] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5107.174455] LustreError: 193887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5107.177504] Lustre: Skipped 2 previous similar messages [ 5107.182537] LustreError: 193887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5112.292496] LustreError: 193888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5112.296754] LustreError: 193888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5121.503851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5121.507218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5121.509836] Lustre: Skipped 1 previous similar message [ 5122.532097] LustreError: 193885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5122.536317] LustreError: 193885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5133.279160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5133.350082] Lustre: server umount lustre-MDT0000 complete [ 5134.596378] LustreError: 193154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771404589 with bad export cookie 10190442947886105405 [ 5134.599126] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5134.600405] LustreError: 193154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5137.889737] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5137.891513] Lustre: Skipped 13 previous similar messages [ 5139.936418] LustreError: 195210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5139.943381] LustreError: 195210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5140.856615] Lustre: server umount lustre-MDT0001 complete [ 5142.150221] Lustre: server umount lustre-OST0000 complete [ 5143.361147] Lustre: server umount lustre-OST0001 complete [ 5145.195491] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 5147.920786] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 5160.737794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5160.833207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5160.846129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5160.885543] Lustre: lustre-MDT0000: new disk, initializing [ 5160.911118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5160.913399] Lustre: Skipped 1 previous similar message [ 5160.919367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5162.376766] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5166.610467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.645326] Lustre: 201914:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5166.747588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5168.039479] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5170.996889] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5172.125291] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5174.283707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5175.670583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5175.702863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5176.238480] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5179.281921] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5180.593381] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5180.667754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5182.027649] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5182.109120] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5186.017504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5186.020770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5186.020908] LustreError: Skipped 1 previous similar message [ 5186.024300] Lustre: Skipped 6 previous similar messages [ 5186.026915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5186.029033] Lustre: Skipped 3 previous similar messages [ 5197.279108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5197.325011] Lustre: server umount lustre-OST0000 complete [ 5204.635984] Lustre: server umount lustre-MDT0000 complete [ 5205.902233] LustreError: 203655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771404660 with bad export cookie 10190442947886107680 [ 5205.905990] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5205.906933] LustreError: 203655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5206.023977] Lustre: server umount lustre-MDT0001 complete [ 5207.257326] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:51:02 (1771404662) [ 5212.595179] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 5215.850340] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 5228.830211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5228.934449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5228.946820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5228.949434] Lustre: Skipped 3 previous similar messages [ 5228.988423] Lustre: lustre-MDT0000: new disk, initializing [ 5228.989986] Lustre: Skipped 2 previous similar messages [ 5229.018133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5229.021960] Lustre: Skipped 2 previous similar messages [ 5229.028843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5229.032157] Lustre: Skipped 2 previous similar messages [ 5230.426212] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5234.632725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5234.680484] Lustre: 208076:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5234.797551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5236.163725] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5239.262961] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5240.519917] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5242.799344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5244.739419] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5244.839661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5247.675247] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5248.977159] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5249.048152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5250.317865] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5250.390953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5255.139018] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.139618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5255.143029] Lustre: Skipped 2 previous similar messages [ 5255.143717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5255.146834] LustreError: Skipped 1 previous similar message [ 5255.152912] Lustre: Skipped 7 previous similar messages [ 5265.375140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5265.433745] Lustre: server umount lustre-OST0000 complete [ 5281.247125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5281.631813] Lustre: server umount lustre-MDT0000 complete [ 5285.314649] LustreError: 209819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771404740 with bad export cookie 10190442947886108562 [ 5285.316607] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5285.334102] LustreError: 209819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5285.609405] Lustre: server umount lustre-MDT0001 complete [ 5293.145372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5293.455243] LustreError: 211889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5296.732969] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5303.494539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5307.750811] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5311.977609] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5314.411314] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5318.207018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5321.091759] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5323.919616] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5349.345812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5349.349279] LustreError: Skipped 1 previous similar message [ 5352.820773] Lustre: server umount lustre-OST0000 complete [ 5360.807283] Lustre: server umount lustre-MDT0000 complete [ 5362.643518] LustreError: 214452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771404817 with bad export cookie 10190442947886109444 [ 5362.646170] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5362.651853] LustreError: 214452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5362.816704] Lustre: server umount lustre-MDT0001 complete [ 5457.002812] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 5458.345695] Key type lgssc unregistered [ 5458.501079] LNet: 215219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5458.504885] LNetError: 215219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5458.512437] LNet: Removed LNI 192.168.203.114@tcp [ 5458.892143] Key type .llcrypt unregistered [ 5458.894078] Key type ._llcrypt unregistered [ 5469.485956] Key type ._llcrypt registered [ 5469.487237] Key type .llcrypt registered [ 5469.534602] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 5476.706968] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 5477.200509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5477.236424] alg: No test for adler32 (adler32-zlib) [ 5478.128529] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 5478.257538] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 5479.859062] Key type lgssc registered [ 5480.433969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5519.952970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5519.980369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.336612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5521.438429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5521.603902] Lustre: lustre-MDT0000: new disk, initializing [ 5521.806208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5521.844221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5527.249265] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5540.842952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.077660] Lustre: 218513:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5541.231699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5541.257056] Lustre: Skipped 1 previous similar message [ 5541.430588] Lustre: lustre-MDT0001: new disk, initializing [ 5541.544375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5541.582544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5541.604289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5547.142375] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5555.416123] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5558.850703] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5564.604320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5564.807449] Lustre: lustre-OST0000: new disk, initializing [ 5564.811341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5564.884565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5570.130712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5570.139561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5570.211113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5570.450884] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5576.928819] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5580.924648] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5581.063067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5584.265595] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5584.441994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5591.011774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.013903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5591.022300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5591.030541] Lustre: Skipped 1 previous similar message [ 5596.132710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5596.145818] Lustre: Skipped 2 previous similar messages [ 5600.223189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5600.398082] Lustre: server umount lustre-OST0000 complete [ 5603.307845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5603.315122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.327214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5606.368864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5606.369096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5609.570015] Lustre: server umount lustre-MDT0000 complete [ 5612.447505] LustreError: 219294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771405067 with bad export cookie 6033939573324034390 [ 5612.455076] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5612.460784] LustreError: 219294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5612.708607] Lustre: server umount lustre-MDT0001 complete [ 5630.039670] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 5639.664511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.098050] LustreError: 223031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.128684] LustreError: 223031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5640.178806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5643.997830] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5645.280847] LustreError: 223031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5650.403847] LustreError: 223031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5652.328414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5652.661292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5656.801758] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5659.545252] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5665.244223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5665.514226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5670.418828] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5677.773118] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5678.006854] Lustre: lustre-OST0001: new disk, initializing [ 5678.016032] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5683.371622] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5686.450212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5686.469714] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5686.604479] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5692.810804] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5698.524813] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5715.779770] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:59:30 (1771405170) [ 5717.474161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5717.491305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5717.492908] Lustre: Skipped 1 previous similar message [ 5717.502227] Lustre: Skipped 2 previous similar messages [ 5723.574582] Lustre: server umount lustre-MDT0000 complete [ 5727.147864] LustreError: 223012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771405182 with bad export cookie 6033939573324035272 [ 5727.157780] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5727.163916] LustreError: 223012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5727.541250] Lustre: server umount lustre-MDT0001 complete [ 5741.705706] Lustre: server umount lustre-OST0000 complete [ 5743.071133] Lustre: 216737:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771405182/real 1771405182] req@ffff8d90c2e54700 x1857452699133184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771405198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5743.113369] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.137462] Lustre: Skipped 2 previous similar messages [ 5745.112353] Lustre: server umount lustre-OST0001 complete [ 5750.069416] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 5757.821883] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 5786.868459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.082754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5787.151343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5787.253387] Lustre: lustre-MDT0000: new disk, initializing [ 5787.416378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5787.421504] Lustre: Skipped 1 previous similar message [ 5787.432374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5791.017470] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5799.489777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.584857] Lustre: 229200:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5799.624850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5799.630135] Lustre: Skipped 1 previous similar message [ 5799.684381] Lustre: lustre-MDT0001: new disk, initializing [ 5799.818437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5799.825823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5803.456614] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5809.811490] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5812.650339] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5818.863740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5819.068994] Lustre: lustre-OST0000: new disk, initializing [ 5819.073225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5820.952042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5820.965607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5821.145554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5824.446335] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5830.253046] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5833.483394] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5833.666568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5837.845385] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5837.987316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5841.384316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5841.388009] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5841.401188] Lustre: Skipped 1 previous similar message [ 5841.405691] Lustre: Skipped 4 previous similar messages [ 5845.839658] Lustre: server umount lustre-OST0000 complete [ 5851.105613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5861.857645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5861.877435] Lustre: Skipped 6 previous similar messages [ 5863.391144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5864.382582] Lustre: server umount lustre-MDT0000 complete [ 5866.985077] LustreError: 230059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5867.024305] LustreError: 230059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5868.462105] LustreError: 229191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771405323 with bad export cookie 6033939573324037414 [ 5868.466161] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5868.478954] LustreError: 229191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5869.027838] Lustre: server umount lustre-MDT0001 complete [ 5878.103388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.409352] LustreError: 233024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5878.459734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5878.464375] Lustre: Skipped 2 previous similar messages [ 5881.726668] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5888.962259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5889.007914] LustreError: 233024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5889.023558] LustreError: 233024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5893.411247] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5898.548806] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5901.343425] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5907.092028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5912.536613] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5917.644559] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6062.054347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6062.062471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6062.063628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6062.065031] Lustre: Skipped 3 previous similar messages [ 6074.847513] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6074.992905] Lustre: server umount lustre-OST0000 complete [ 6078.947976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6093.280791] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6093.547734] Lustre: server umount lustre-MDT0000 complete [ 6096.452508] LustreError: 234639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771405551 with bad export cookie 6033939573324038296 [ 6096.454132] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6096.460878] LustreError: 234639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6096.762740] Lustre: server umount lustre-MDT0001 complete [ 6183.787398] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 6185.701486] Key type lgssc unregistered [ 6185.940747] LNet: 236279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6185.947831] LNetError: 236279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6185.960601] LNet: Removed LNI 192.168.203.114@tcp [ 6186.608177] Key type .llcrypt unregistered [ 6186.611820] Key type ._llcrypt unregistered [ 6207.569733] Key type ._llcrypt registered [ 6207.571969] Key type .llcrypt registered [ 6207.716520] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 6223.982772] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6225.266319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6225.306862] alg: No test for adler32 (adler32-zlib) [ 6226.322878] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 6226.466688] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 6228.103176] Key type lgssc registered [ 6229.279824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6271.755723] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6282.163708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6282.207472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6283.519293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6283.571798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6283.684566] Lustre: lustre-MDT0000: new disk, initializing [ 6283.792925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6283.837334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6288.161217] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6298.152461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6298.288787] Lustre: 240299:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6298.349375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6298.360394] Lustre: Skipped 1 previous similar message [ 6298.462493] Lustre: lustre-MDT0001: new disk, initializing [ 6298.566975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6298.625041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6298.639419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6302.266224] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6306.569514] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6312.446813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6312.754607] Lustre: lustre-OST0000: new disk, initializing [ 6312.758618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6312.856200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6317.768335] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6320.197023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6320.210359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6320.332562] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6327.752913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6327.966484] Lustre: lustre-OST0001: new disk, initializing [ 6327.970344] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6328.089478] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6333.710916] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6334.076016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6334.094978] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6334.195068] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6344.076248] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6352.395936] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6370.279576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6370.284438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6370.305322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6370.313496] Lustre: Skipped 3 previous similar messages [ 6370.399074] Lustre: server umount lustre-MDT0000 complete [ 6374.078555] LustreError: 240290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771405829 with bad export cookie 12932635049272250022 [ 6374.088196] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6374.093090] LustreError: 240290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6374.595123] Lustre: server umount lustre-MDT0001 complete [ 6389.039911] Lustre: server umount lustre-OST0000 complete [ 6391.264745] Lustre: 237823:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771405830/real 1771405830] req@ffff8d91cfac0a80 x1857453483437568/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771405846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6391.288654] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6391.296583] Lustre: Skipped 3 previous similar messages [ 6392.783937] Lustre: server umount lustre-OST0001 complete [ 6403.201287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6411.772608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6419.119855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6427.634398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6443.518526] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6451.683837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6451.704197] Lustre: lustre-MDT0000: reset Object Index mappings [ 6452.056584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6455.763988] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6457.320927] LustreError: 247724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6457.341193] LustreError: 247724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6461.411456] LustreError: 247724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6463.582816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.617095] Lustre: lustre-MDT0001: reset Object Index mappings [ 6464.148258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6467.721346] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6469.916423] Lustre: 248829:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6475.856172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6477.160327] LustreError: 249185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6477.190065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6480.836571] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6482.413421] LustreError: 249185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6487.521395] LustreError: 249185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6488.633485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6488.881128] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6488.888246] Lustre: Skipped 1 previous similar message [ 6490.936920] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6494.279884] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6500.984687] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6504.043831] Lustre: 250673:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6514.660328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6514.662593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6514.675204] Lustre: Skipped 3 previous similar messages [ 6514.687114] Lustre: Skipped 3 previous similar messages [ 6518.934357] Lustre: server umount lustre-MDT0000 complete [ 6519.778696] LustreError: 250317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6519.804977] LustreError: 250317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6522.425112] LustreError: 247703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771405977 with bad export cookie 12932635049272256812 [ 6522.435749] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6522.767788] Lustre: server umount lustre-MDT0001 complete [ 6535.242328] Lustre: server umount lustre-OST0000 complete [ 6548.861952] Lustre: server umount lustre-OST0001 complete [ 6565.617773] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 04:13:39 (1771406019) [ 6579.566610] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 6586.890864] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6618.737364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.070859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6619.101938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6619.188086] Lustre: lustre-MDT0000: new disk, initializing [ 6619.271597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6619.288979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6623.377429] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6633.595517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6633.748928] Lustre: 254896:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6633.804537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6633.808348] Lustre: Skipped 1 previous similar message [ 6633.907304] Lustre: lustre-MDT0001: new disk, initializing [ 6634.108044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6634.126284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6638.294264] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6646.479776] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6650.747672] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6657.867759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6658.119861] Lustre: lustre-OST0000: new disk, initializing [ 6658.123188] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6659.326386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6659.339703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6659.498458] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6663.889917] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6670.771338] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6674.945365] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6675.137603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6678.797683] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6678.967492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6685.153442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6685.156828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6685.180801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6685.198529] Lustre: Skipped 1 previous similar message [ 6686.900748] Lustre: server umount lustre-OST0000 complete [ 6690.272705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6690.278655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6690.298249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6690.305636] Lustre: Skipped 2 previous similar messages [ 6695.393975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6695.397054] Lustre: Skipped 2 previous similar messages [ 6696.334173] Lustre: server umount lustre-MDT0000 complete [ 6700.302307] LustreError: 254888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771406155 with bad export cookie 12932635049272258905 [ 6700.308968] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6700.313758] LustreError: 254888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6700.783099] Lustre: server umount lustre-MDT0001 complete [ 6717.060880] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6725.016825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6725.283602] LustreError: 259404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6725.315435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6725.318391] Lustre: Skipped 2 previous similar messages [ 6728.705777] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6735.905427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6739.530864] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6741.749573] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6747.429215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6753.113917] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6760.786340] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6760.912548] Lustre: lustre-OST0001: new disk, initializing [ 6760.917858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6762.285388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6762.293289] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6762.377142] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6765.916637] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6774.038355] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6777.239515] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6786.883175] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6786.924737] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6792.674271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6792.679064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6792.695849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6794.321667] Lustre: server umount lustre-MDT0000 complete [ 6797.718826] LustreError: 259383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771406252 with bad export cookie 12932635049272259787 [ 6797.722952] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6797.733085] LustreError: 259383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6797.970276] Lustre: server umount lustre-MDT0001 complete [ 6810.350099] Lustre: server umount lustre-OST0000 complete [ 6824.195411] Lustre: server umount lustre-OST0001 complete [ 6836.354583] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6844.789077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6845.105700] LustreError: 264683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6845.119221] LustreError: 264683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6849.418483] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6857.092716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6857.396976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6857.401871] Lustre: Skipped 4 previous similar messages [ 6860.623426] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6863.479781] Lustre: 265790:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6869.114650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6870.388259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6874.612689] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6881.225098] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6882.518955] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6886.341527] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6892.412262] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6895.671439] Lustre: 267641:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6905.491535] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6905.506850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6913.669336] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 04:19:28 (1771406368) [ 6928.868295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6928.886375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6928.887064] Lustre: Skipped 5 previous similar messages [ 6928.905830] Lustre: Skipped 7 previous similar messages [ 6930.298434] Lustre: server umount lustre-MDT0000 complete [ 6933.836510] LustreError: 264663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771406388 with bad export cookie 12932635049272266101 [ 6933.855494] LustreError: 264663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6933.865419] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6933.988055] LustreError: 264679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6933.996606] LustreError: 264679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6948.319759] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6948.579423] Lustre: server umount lustre-MDT0001 complete [ 6966.240095] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6966.414954] Lustre: server umount lustre-OST0000 complete [ 6969.223501] Lustre: server umount lustre-OST0001 complete [ 6974.043978] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 6981.733808] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7016.959196] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7024.516952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.821620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7024.867750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7024.968651] Lustre: lustre-MDT0000: new disk, initializing [ 7025.088795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7028.865533] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7037.318949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7037.461314] Lustre: 271828:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7037.652285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7040.831785] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7050.080965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7051.667272] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7055.695241] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7064.151126] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7066.322553] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7068.514848] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7076.225149] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7078.926210] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7078.929271] Lustre: Skipped 1 previous similar message [ 7094.151362] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 04:22:28 (1771406548) [ 7097.314962] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7097.324561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7097.331759] Lustre: Skipped 6 previous similar messages [ 7097.337384] Lustre: Skipped 10 previous similar messages [ 7101.991869] Lustre: server umount lustre-MDT0000 complete [ 7102.433471] LustreError: 271837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7102.452431] LustreError: 271837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 7104.736366] LustreError: 271823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771406559 with bad export cookie 12932635049272273962 [ 7104.738120] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7104.741807] LustreError: 271823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7104.997991] Lustre: server umount lustre-MDT0001 complete [ 7117.883299] Lustre: server umount lustre-OST0000 complete [ 7131.048361] Lustre: server umount lustre-OST0001 complete [ 7144.967589] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7152.016852] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7175.144207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7175.321659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7175.342137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7175.347058] Lustre: Skipped 4 previous similar messages [ 7175.433247] Lustre: lustre-MDT0000: new disk, initializing [ 7175.438437] Lustre: Skipped 3 previous similar messages [ 7175.537216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7175.543423] Lustre: Skipped 6 previous similar messages [ 7175.557304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7175.560476] Lustre: Skipped 3 previous similar messages [ 7178.271364] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7185.163950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7185.258927] Lustre: 278865:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7185.450071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7185.455579] Lustre: Skipped 2 previous similar messages [ 7187.870714] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7192.579622] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7195.119214] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7199.490576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7201.058507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7203.514662] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7208.957673] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7210.292335] Lustre: Mounted lustre-client [ 7210.829587] LustreError: 281361:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8d91c40f3000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 7210.914284] Lustre: Unmounted lustre-client [ 7216.612056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7216.615365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7216.615708] LustreError: Skipped 1 previous similar message [ 7216.629565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7216.634577] Lustre: Skipped 2 previous similar messages [ 7222.597737] Lustre: server umount lustre-OST0000 complete [ 7239.135144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7239.289409] Lustre: server umount lustre-MDT0000 complete [ 7241.616615] LustreError: 278856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771406696 with bad export cookie 12932635049272275495 [ 7241.620176] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7241.624867] LustreError: 278856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7241.834867] Lustre: server umount lustre-MDT0001 complete [ 7247.694334] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7249.415793] Key type lgssc unregistered [ 7265.823143] LNet: 1 local NIs in recovery (showing 1): 192.168.203.114@tcp [ 7266.016598] LNet: 282640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7266.025626] LNetError: 282640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7341.800252] LNet: Removed LNI 192.168.203.114@tcp [ 7342.322144] Key type .llcrypt unregistered [ 7342.323741] Key type ._llcrypt unregistered [ 7352.416874] Key type ._llcrypt registered [ 7352.419104] Key type .llcrypt registered [ 7352.470303] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7365.320649] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7366.024333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7366.049414] alg: No test for adler32 (adler32-zlib) [ 7367.079608] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 7367.239829] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 7368.888864] Key type lgssc registered [ 7369.881086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7388.920367] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:27:23 (1771406843) [ 7392.438200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7392.450299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7393.572887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7393.589637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7393.630319] Lustre: lustre-MDT0000: new disk, initializing [ 7393.677171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7393.690857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7395.742521] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7401.328151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7401.397616] Lustre: 285829:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7401.450256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7401.453402] Lustre: Skipped 1 previous similar message [ 7401.512689] Lustre: lustre-MDT0001: new disk, initializing [ 7401.599625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7401.620923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7401.625673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7403.266344] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7407.149327] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7408.960293] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7412.108905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7412.254342] Lustre: lustre-OST0000: new disk, initializing [ 7412.256797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7412.305124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7415.130421] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7417.858307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7417.863060] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7417.924735] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7418.945466] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7428.066361] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7428.067236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7428.067929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7428.070087] LustreError: Skipped 1 previous similar message [ 7428.074367] Lustre: Skipped 1 previous similar message [ 7433.192701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7433.195119] Lustre: Skipped 2 previous similar messages [ 7438.303157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7438.305354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7438.310432] Lustre: Skipped 1 previous similar message [ 7438.392973] Lustre: server umount lustre-OST0000 complete [ 7442.399971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7442.402720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7442.408262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7443.425607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7446.376867] Lustre: server umount lustre-MDT0000 complete [ 7447.779295] LustreError: 285821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771406902 with bad export cookie 7270634180592038570 [ 7447.781775] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7447.783370] LustreError: 285821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7447.920664] Lustre: server umount lustre-MDT0001 complete [ 7452.149642] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7453.273140] Key type lgssc unregistered [ 7453.404371] LNet: 289198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7453.407824] LNetError: 289198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7453.415461] LNet: Removed LNI 192.168.203.114@tcp [ 7453.736789] Key type .llcrypt unregistered [ 7453.738132] Key type ._llcrypt unregistered [ 7465.219510] Key type ._llcrypt registered [ 7465.221011] Key type .llcrypt registered [ 7465.269393] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7465.640148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7465.652311] alg: No test for adler32 (adler32-zlib) [ 7466.535355] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 7466.653016] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 7468.256265] Key type lgssc registered [ 7468.803620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7472.108414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7472.120713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7473.325057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7475.005370] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7479.295970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7479.483392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7481.402982] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7484.203268] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7486.031930] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7489.339746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7489.498293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7492.240671] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7494.893781] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7503.982705] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7504.855873] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7508.966772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7508.967846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7508.979701] Lustre: Skipped 2 previous similar messages [ 7508.989163] Lustre: Skipped 2 previous similar messages [ 7510.780218] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.14@tcp (stopping) [ 7512.802863] Lustre: server umount lustre-MDT0000 complete [ 7514.083310] LustreError: 290574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7514.095852] LustreError: 290574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7514.476631] LustreError: 290554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771406969 with bad export cookie 5093408215271148970 [ 7514.481167] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7514.483178] LustreError: 290554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7515.891125] LustreError: 290570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7515.904929] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.14@tcp (stopping) [ 7519.200185] LustreError: 290571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7519.200428] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7519.212069] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7520.694364] Lustre: server umount lustre-MDT0001 complete [ 7528.224528] Lustre: server umount lustre-OST0000 complete [ 7531.764273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7533.270307] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7534.400389] Lustre: 294302:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.114@tcp [ 7535.378496] Lustre: server umount MGS complete [ 7539.396627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7539.644598] LustreError: 294874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7539.697377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7541.787105] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7544.800141] LustreError: 294874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7546.238357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7546.417133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7548.100664] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7551.999068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7554.931899] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7557.409381] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7558.158194] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7559.707281] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7568.628275] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 7569.890472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7569.890965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7569.893500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7569.897389] Lustre: Skipped 1 previous similar message [ 7569.901325] Lustre: Skipped 1 previous similar message [ 7580.129488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7580.132323] Lustre: Skipped 3 previous similar messages [ 7584.223143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7584.318070] Lustre: server umount lustre-OST0000 complete [ 7587.296495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7587.303602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7592.312523] Lustre: server umount lustre-MDT0000 complete [ 7593.824403] LustreError: 297030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771407048 with bad export cookie 5093408215271149810 [ 7593.828836] LustreError: 297030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7593.829778] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7593.954455] Lustre: server umount lustre-MDT0001 complete [ 7598.250227] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7599.432382] Key type lgssc unregistered [ 7599.578533] LNet: 298054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7599.581616] LNetError: 298054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7599.592036] LNet: Removed LNI 192.168.203.114@tcp [ 7599.951042] Key type .llcrypt unregistered [ 7599.953090] Key type ._llcrypt unregistered [ 7607.006959] Key type ._llcrypt registered [ 7607.008585] Key type .llcrypt registered [ 7607.051415] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7613.766175] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7614.186466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7614.222673] alg: No test for adler32 (adler32-zlib) [ 7615.096309] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 7615.189266] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 7616.783182] Key type lgssc registered [ 7617.223385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7632.558195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7632.567505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7633.679139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7633.695974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7633.733251] Lustre: lustre-MDT0000: new disk, initializing [ 7633.762131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7633.770770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7635.376777] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7640.246698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7640.298687] Lustre: 301152:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7640.324517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7640.327031] Lustre: Skipped 1 previous similar message [ 7640.366118] Lustre: lustre-MDT0001: new disk, initializing [ 7640.409428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7640.423269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7640.427849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7642.193689] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7645.999762] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7647.349982] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7649.978484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7650.094271] Lustre: lustre-OST0000: new disk, initializing [ 7650.098904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7650.139513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7651.646602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7651.651222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7651.690079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7652.257978] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7655.890973] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7657.534298] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7657.618879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7659.042154] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7659.134987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7662.049146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7662.050278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7662.052036] LustreError: Skipped 1 previous similar message [ 7662.057022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7662.061320] Lustre: Skipped 1 previous similar message [ 7667.170161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7667.174801] Lustre: Skipped 1 previous similar message [ 7672.289257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7672.291779] Lustre: Skipped 1 previous similar message [ 7674.335329] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7674.400526] Lustre: server umount lustre-OST0000 complete [ 7676.384522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7676.390622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7676.395759] Lustre: Skipped 1 previous similar message [ 7676.398169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7677.409733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.146459] Lustre: server umount lustre-MDT0000 complete [ 7683.679331] LustreError: 301144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771407138 with bad export cookie 2050948604632971595 [ 7683.682468] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7683.684146] LustreError: 301144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7683.795485] Lustre: server umount lustre-MDT0001 complete [ 7687.025530] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 04:32:22 (1771407142) [ 7706.228472] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 04:32:41 (1771407161) [ 7723.905528] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7725.400265] Key type lgssc unregistered [ 7725.578585] LNet: 307696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7725.582416] LNetError: 307696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7725.595147] LNet: Removed LNI 192.168.203.114@tcp [ 7726.040066] Key type .llcrypt unregistered [ 7726.042373] Key type ._llcrypt unregistered [ 7773.110922] Key type ._llcrypt registered [ 7773.112945] Key type .llcrypt registered [ 7773.187591] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7773.370047] Key type .llcrypt unregistered [ 7773.374048] Key type ._llcrypt unregistered [ 7789.564929] Key type ._llcrypt registered [ 7789.569085] Key type .llcrypt registered [ 7789.625710] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7789.771498] Key type .llcrypt unregistered [ 7789.772858] Key type ._llcrypt unregistered [ 7803.965247] Key type ._llcrypt registered [ 7803.966754] Key type .llcrypt registered [ 7804.013202] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7804.147384] Key type .llcrypt unregistered [ 7804.148394] Key type ._llcrypt unregistered [ 7841.901860] Key type ._llcrypt registered [ 7841.903121] Key type .llcrypt registered [ 7841.964724] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7842.113353] Key type .llcrypt unregistered [ 7842.115024] Key type ._llcrypt unregistered [ 7856.953300] Key type ._llcrypt registered [ 7856.954782] Key type .llcrypt registered [ 7857.004837] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7857.129498] Key type .llcrypt unregistered [ 7857.131423] Key type ._llcrypt unregistered [ 7870.971391] Key type ._llcrypt registered [ 7870.972404] Key type .llcrypt registered [ 7871.032067] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7871.189569] Key type .llcrypt unregistered [ 7871.191453] Key type ._llcrypt unregistered [ 7885.803100] Key type ._llcrypt registered [ 7885.804455] Key type .llcrypt registered [ 7885.854700] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7886.035635] Key type .llcrypt unregistered [ 7886.037345] Key type ._llcrypt unregistered [ 7901.712967] Key type ._llcrypt registered [ 7901.714154] Key type .llcrypt registered [ 7901.787261] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7901.941433] Key type .llcrypt unregistered [ 7901.944476] Key type ._llcrypt unregistered [ 7915.447219] Key type ._llcrypt registered [ 7915.448918] Key type .llcrypt registered [ 7915.508177] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7915.651302] Key type .llcrypt unregistered [ 7915.653228] Key type ._llcrypt unregistered [ 7946.574404] Key type ._llcrypt registered [ 7946.576092] Key type .llcrypt registered [ 7946.630821] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7946.792533] Key type .llcrypt unregistered [ 7946.796072] Key type ._llcrypt unregistered [ 7961.739178] Key type ._llcrypt registered [ 7961.740894] Key type .llcrypt registered [ 7961.784631] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7961.917494] Key type .llcrypt unregistered [ 7961.918760] Key type ._llcrypt unregistered [ 7975.758853] Key type ._llcrypt registered [ 7975.760758] Key type .llcrypt registered [ 7975.818461] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7975.958407] Key type .llcrypt unregistered [ 7975.960075] Key type ._llcrypt unregistered [ 8003.312271] Key type ._llcrypt registered [ 8003.315221] Key type .llcrypt registered [ 8003.372920] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 8010.871082] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8011.239935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8011.295144] alg: No test for adler32 (adler32-zlib) [ 8012.170885] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 8012.264301] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8013.855179] Key type lgssc registered [ 8014.284423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8027.699192] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7733 sec ========== 04:38:02 (1771407482) [ 8028.463421] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:38:03 (1771407483) === [ 8029.925924] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:38:04 (1771407484) === [ 8046.125191] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8047.295628] Key type lgssc unregistered [ 8047.426474] LNet: 326518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8047.430140] LNetError: 326518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8047.440370] LNet: Removed LNI 192.168.203.114@tcp [ 8047.744672] Key type .llcrypt unregistered [ 8047.745982] Key type ._llcrypt unregistered