[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 429978286 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003254] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005044] kvm-guest: setup PV IPIs [ 0.008895] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011125] LSM: Security Framework initializing [ 0.012071] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014109] *** VALIDATE selinux *** [ 0.021694] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025680] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026187] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028040] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029134] *** VALIDATE tmpfs *** [ 0.031073] *** VALIDATE proc *** [ 0.032191] *** VALIDATE cgroup *** [ 0.033017] *** VALIDATE cgroup2 *** [ 0.035223] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042435] debug: unmapping init [mem 0xffffffffb4459000-0xffffffffb4460fff] [ 0.044159] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045769] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048008] ... generic registers: 4 [ 0.049017] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051008] ... fixed-purpose events: 3 [ 0.052019] ... event mask: 000000070000000f [ 0.053286] rcu: Hierarchical SRCU implementation. [ 0.055461] smp: Bringing up secondary CPUs ... [ 0.056572] x86: Booting SMP configuration: [ 0.057024] .... node #0, CPUs: #1 #2 #3 [ 0.060200] smp: Brought up 1 node, 4 CPUs [ 0.062010] smpboot: Max logical packages: 1 [ 0.063017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.160872] node 0 deferred pages initialised in 95ms [ 0.164093] devtmpfs: initialized [ 0.165248] x86/mm: Memory block size: 128MB [ 0.167731] gcov: version magic: 0x41383552 [ 0.171086] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.174103] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.176292] pinctrl core: initialized pinctrl subsystem [ 0.178152] [ 0.178724] ************************************************************* [ 0.181013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183009] ** ** [ 0.185009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.187014] ** ** [ 0.189014] ** This means that this kernel is built to expose internal ** [ 0.192012] ** IOMMU data structures, which may compromise security on ** [ 0.194010] ** your system. ** [ 0.196013] ** ** [ 0.198014] ** If you see this message and you are not debugging the ** [ 0.200010] ** kernel, report this immediately to your vendor! ** [ 0.202010] ** ** [ 0.204009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.207011] ************************************************************* [ 0.209874] NET: Registered protocol family 16 [ 0.211544] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.214081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.215062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.218124] cpuidle: using governor menu [ 0.219728] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.221525] PCI: Using configuration type 1 for base access [ 0.224140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.232135] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.233030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.234205] cryptd: max_cpu_qlen set to 1000 [ 0.238354] ACPI: Added _OSI(Module Device) [ 0.240021] ACPI: Added _OSI(Processor Device) [ 0.241010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.242007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.245150] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.251487] ACPI: Interpreter enabled [ 0.253061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.254012] ACPI: Using IOAPIC for interrupt routing [ 0.256172] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.258443] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.268448] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276102] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281466] acpiphp: Slot [2] registered [ 0.282124] acpiphp: Slot [5] registered [ 0.284123] acpiphp: Slot [6] registered [ 0.285066] acpiphp: Slot [7] registered [ 0.286095] acpiphp: Slot [8] registered [ 0.287070] acpiphp: Slot [9] registered [ 0.288053] acpiphp: Slot [10] registered [ 0.290123] acpiphp: Slot [3] registered [ 0.291109] acpiphp: Slot [4] registered [ 0.292067] acpiphp: Slot [11] registered [ 0.294092] acpiphp: Slot [12] registered [ 0.295058] acpiphp: Slot [13] registered [ 0.296073] acpiphp: Slot [14] registered [ 0.297053] acpiphp: Slot [15] registered [ 0.298053] acpiphp: Slot [16] registered [ 0.300105] acpiphp: Slot [17] registered [ 0.301063] acpiphp: Slot [18] registered [ 0.302062] acpiphp: Slot [19] registered [ 0.304077] acpiphp: Slot [20] registered [ 0.305072] acpiphp: Slot [21] registered [ 0.306051] acpiphp: Slot [22] registered [ 0.307127] acpiphp: Slot [23] registered [ 0.309102] acpiphp: Slot [24] registered [ 0.310074] acpiphp: Slot [25] registered [ 0.312066] acpiphp: Slot [26] registered [ 0.313066] acpiphp: Slot [27] registered [ 0.314051] acpiphp: Slot [28] registered [ 0.315075] acpiphp: Slot [29] registered [ 0.315880] acpiphp: Slot [30] registered [ 0.316070] acpiphp: Slot [31] registered [ 0.318179] PCI host bridge to bus 0000:00 [ 0.320020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.323023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.325026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.328024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330011] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.332029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333160] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334732] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.338175] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.347016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.351540] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.353020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.356030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.359021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.361600] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.364440] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.367039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.369526] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.373012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.382022] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.386016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.392293] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.397011] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.402018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.416018] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.427529] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.433022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.439037] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.456039] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.466960] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.474022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.479019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.493019] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.506134] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.512019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.517038] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.534019] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.543734] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.552015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.558023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.571020] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.582596] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.589026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.596027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.613032] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.624312] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.627510] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.629426] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.632325] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.634160] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.638131] iommu: Default domain type: Passthrough [ 0.640533] SCSI subsystem initialized [ 0.642129] ACPI: bus type USB registered [ 0.643117] usbcore: registered new interface driver usbfs [ 0.645141] usbcore: registered new interface driver hub [ 0.648168] usbcore: registered new device driver usb [ 0.650179] pps_core: LinuxPPS API ver. 1 registered [ 0.652008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.654157] PTP clock support registered [ 0.657312] EDAC MC: Ver: 3.0.0 [ 0.659130] PCI: Using ACPI for IRQ routing [ 0.660595] NetLabel: Initializing [ 0.662017] NetLabel: domain hash size = 128 [ 0.663009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.665077] NetLabel: unlabeled traffic allowed by default [ 0.667259] vgaarb: loaded [ 0.669390] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.671017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.676006] clocksource: Switched to clocksource kvm-clock [ 0.788378] VFS: Disk quotas dquot_6.6.0 [ 0.790100] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.793025] *** VALIDATE ramfs *** [ 0.794609] *** VALIDATE hugetlbfs *** [ 0.796476] pnp: PnP ACPI init [ 0.799936] pnp: PnP ACPI: found 6 devices [ 0.813841] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.816099] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.817672] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.819011] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.821456] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.823875] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.826821] NET: Registered protocol family 2 [ 0.829167] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.832952] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.836529] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.841432] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.844766] TCP: Hash tables configured (established 65536 bind 65536) [ 0.847588] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.850683] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.853644] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.856845] NET: Registered protocol family 1 [ 0.859511] RPC: Registered named UNIX socket transport module. [ 0.861559] RPC: Registered udp transport module. [ 0.863169] RPC: Registered tcp transport module. [ 0.864768] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.866988] NET: Registered protocol family 44 [ 0.868523] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.870726] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.872732] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.875066] PCI: CLS 0 bytes, default 64 [ 0.876704] Unpacking initramfs... [ 2.576279] debug: unmapping init [mem 0xffff9a2afcc54000-0xffff9a2afffbffff] [ 2.585747] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.588464] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.591789] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.190692] Initialise system trusted keyrings [ 3.192142] Key type blacklist registered [ 3.195229] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.205411] zbud: loaded [ 3.209381] *** VALIDATE nfs *** [ 3.210519] *** VALIDATE nfs4 *** [ 3.212031] pstore: using deflate compression [ 3.215568] Platform Keyring initialized [ 3.366276] NET: Registered protocol family 38 [ 3.367334] Key type asymmetric registered [ 3.368255] Asymmetric key parser 'x509' registered [ 3.371334] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.375049] io scheduler mq-deadline registered [ 3.377627] io scheduler kyber registered [ 3.381636] io scheduler bfq registered [ 3.383796] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.387293] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.390201] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.393116] ACPI: Power Button [PWRF] [ 3.504817] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.608769] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.877275] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.979200] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.187871] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.223372] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.262941] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.268847] Non-volatile memory driver v1.3 [ 4.270933] Linux agpgart interface v0.103 [ 4.312577] virtio_blk virtio1: [vda] 133424 512-byte logical blocks (68.3 MB/65.1 MiB) [ 4.318225] vda: detected capacity change from 0 to 68313088 [ 4.338596] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.341988] vdb: detected capacity change from 0 to 1073741824 [ 4.360481] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.363476] vdc: detected capacity change from 0 to 2621440000 [ 4.382529] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.386912] vdd: detected capacity change from 0 to 2621440000 [ 4.403948] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.407292] vde: detected capacity change from 0 to 4294967296 [ 4.429278] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.432597] vdf: detected capacity change from 0 to 4294967296 [ 4.441181] libphy: Fixed MDIO Bus: probed [ 4.448728] usbcore: registered new interface driver usbserial_generic [ 4.450954] usbserial: USB Serial support registered for generic [ 4.453112] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.457922] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.460880] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.464101] mousedev: PS/2 mouse device common for all mice [ 4.467962] rtc_cmos 00:05: RTC can wake from S4 [ 4.471788] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.476785] rtc_cmos 00:05: registered as rtc0 [ 4.478672] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.483820] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.485475] intel_pstate: CPU model not supported [ 4.493690] hid: raw HID events driver (C) Jiri Kosina [ 4.494684] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.496959] usbcore: registered new interface driver usbhid [ 4.500532] usbhid: USB HID core driver [ 4.501921] drop_monitor: Initializing network drop monitor service [ 4.504279] Initializing XFRM netlink socket [ 4.505926] NET: Registered protocol family 10 [ 4.508678] Segment Routing with IPv6 [ 4.510029] NET: Registered protocol family 17 [ 4.511930] mpls_gso: MPLS GSO support [ 4.516469] RAS: Correctable Errors collector initialized. [ 4.518153] AVX version of gcm_enc/dec engaged. [ 4.519428] AES CTR mode by8 optimization enabled [ 4.619628] sched_clock: Marking stable (4619423971, 0)->(5387583957, -768159986) [ 4.623831] registered taskstats version 1 [ 4.625778] Loading compiled-in X.509 certificates [ 4.629075] zswap: loaded using pool lzo/zbud [ 4.680819] Key type big_key registered [ 4.698354] Key type encrypted registered [ 4.701529] ima: No TPM chip found, activating TPM-bypass! [ 4.704162] ima: Allocated hash algorithm: sha1 [ 4.706094] ima: No architecture policies found [ 4.707925] evm: Initialising EVM extended attributes: [ 4.709275] evm: security.selinux [ 4.710316] evm: security.ima [ 4.711626] evm: security.capability [ 4.713102] evm: HMAC attrs: 0x1 [ 4.717078] rtc_cmos 00:05: setting system clock to 2025-10-22 13:51:24 UTC (1761141084) [ 4.726103] debug: unmapping init [mem 0xffffffffb5403000-0xffffffffb55fffff] [ 4.736962] debug: unmapping init [mem 0xffffffffb4182000-0xffffffffb4458fff] [ 4.741167] Write protecting the kernel read-only data: 28672k [ 4.744983] debug: unmapping init [mem 0xffffffffb2803000-0xffffffffb29fffff] [ 4.748570] debug: unmapping init [mem 0xffffffffb3114000-0xffffffffb31fffff] [ 4.804279] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.815803] systemd[1]: Detected virtualization kvm. [ 4.818409] systemd[1]: Detected architecture x86-64. [ 4.820665] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.850496] systemd[1]: No hostname configured. [ 4.853675] systemd[1]: Set hostname to . [ 4.855305] random: systemd: uninitialized urandom read (16 bytes read) [ 4.857436] systemd[1]: Initializing machine ID from random generator. [ 4.909169] random: ln: uninitialized urandom read (6 bytes read) [ 5.057447] random: systemd: uninitialized urandom read (16 bytes read) [ 5.061645] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.073294] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 5.095258] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.137350] device-mapper: uevent: version 1.0.3 [ 6.139696] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 6.647243] hrtimer: interrupt took 9748828 ns [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.043233] virtio_net virtio0 ens2: renamed from eth0 [ 8.140130] random: fast init done [ 8.213273] scsi host0: ata_piix [ 8.235649] scsi host1: ata_piix [ 8.238473] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.243012] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.306351] random: crng init done [ 13.308549] random: 7 urandom warning(s) missed due to ratelimiting [ 16.182392] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.742209] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.528990] printk: systemd: 25 output lines suppressed due to ratelimiting [ 19.983040] SELinux: Disabled at runtime. [ 20.111289] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.124771] systemd[1]: Detected virtualization kvm. [ 20.127062] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.146124] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.154215] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.167078] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.173816] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.185526] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.209648] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.243804] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK [[ 21.376317] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 22.421729] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.116403] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.154886] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.607421] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.663570] EDAC sbridge: Ver: 1.1.2 [ 26.789690] Key type dns_resolver registered [ 27.155246] NFS: Registering the id_resolver key type [ 27.157286] Key type id_resolver registered [ 27.158893] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg109-server login: [ 53.802550] libcfs: loading out-of-tree module taints kernel. [ 53.816104] Key type ._llcrypt registered [ 53.817530] Key type .llcrypt registered [ 53.861783] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 61.009779] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 61.697214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.706120] alg: No test for adler32 (adler32-zlib) [ 62.665255] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 62.982901] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 64.592119] Key type lgssc registered [ 65.174300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 71.903758] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 73.216259] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.061606] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.721240] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.472785] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.692086] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 92.489074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.506349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 92.516348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 93.610880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 93.624125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 93.664720] Lustre: lustre-MDT0000: new disk, initializing [ 93.694958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.703774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 94.988913] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 100.312284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 100.339876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 100.366348] Lustre: 6485:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 100.380740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 100.383570] Lustre: Skipped 1 previous similar message [ 100.426095] Lustre: lustre-MDT0001: new disk, initializing [ 100.448850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 100.456955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 100.460439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 101.823725] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 104.041104] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 107.240992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.263303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 107.348194] Lustre: lustre-OST0000: new disk, initializing [ 107.349973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 107.365708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 107.754321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 107.757072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 107.776519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 109.151157] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 114.676910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 114.702921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 114.740929] Lustre: lustre-OST0001: new disk, initializing [ 114.743800] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 114.760694] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 115.513358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 115.516477] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 115.539140] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 116.837352] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 123.273584] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 128.025389] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 135.185879] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing check_logdir /tmp/testlogs/ [ 136.738196] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing yml_node [ 138.136406] Lustre: DEBUG MARKER: Client: 2.16.59.1 [ 139.032283] Lustre: DEBUG MARKER: MDS: 2.16.59.1 [ 139.998424] Lustre: DEBUG MARKER: OSS: 2.16.59.1 [ 140.579993] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Oct 22 09:53:40 EDT 2025 [ 147.469639] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 148.059518] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 161.760982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 161.761576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 161.766271] Lustre: Skipped 2 previous similar messages [ 161.768617] Lustre: Skipped 1 previous similar message [ 166.154267] LustreError: 12190:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 166.196446] Lustre: server umount lustre-MDT0000 complete [ 167.955329] LustreError: 7417:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141247 with bad export cookie 18234293640179553856 [ 167.955577] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 167.960539] LustreError: 7417:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 172.000976] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 172.001134] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 172.001845] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 172.001851] Lustre: Skipped 2 previous similar messages [ 172.007297] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 172.015281] Lustre: Skipped 2 previous similar messages [ 173.024863] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 173.024880] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 173.030584] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 173.032233] Lustre: Skipped 2 previous similar messages [ 178.144614] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 178.144662] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 178.151849] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 178.154609] Lustre: Skipped 1 previous similar message [ 182.240173] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 182.275758] LustreError: 12394:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 182.279352] LustreError: 12394:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 182.387821] Lustre: server umount lustre-MDT0001 complete [ 184.063084] LustreError: 12596:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 184.065598] LustreError: 12596:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 184.088835] Lustre: server umount lustre-OST0000 complete [ 185.886433] Lustre: server umount lustre-OST0001 complete [ 188.889446] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 192.759401] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 197.823275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 200.995390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 203.787438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 206.663235] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 209.671162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 209.702786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.809828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.825441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.875656] Lustre: lustre-MDT0000: new disk, initializing [ 209.914516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.924296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 211.669205] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 216.916372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 216.957648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 217.007822] Lustre: 14974:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 217.181830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 219.028711] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 223.284918] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 224.958434] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 228.119980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 228.151294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 228.290910] Lustre: lustre-OST0000: new disk, initializing [ 228.293098] Lustre: Skipped 1 previous similar message [ 228.294917] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 228.298177] Lustre: Skipped 2 previous similar messages [ 228.333350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 228.336404] Lustre: Skipped 1 previous similar message [ 230.280995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.286110] Lustre: Skipped 1 previous similar message [ 230.288821] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 230.358068] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 230.997369] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 234.839219] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 236.605438] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 236.716264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 238.458691] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 238.551042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 240.612695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 240.613129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 240.613822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 240.619496] Lustre: Skipped 1 previous similar message [ 240.622859] LustreError: Skipped 1 previous similar message [ 245.360095] LustreError: 17913:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 245.362799] LustreError: 17913:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 245.445078] Lustre: server umount lustre-OST0000 complete [ 247.777080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 247.780540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.849625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 250.852684] Lustre: Skipped 3 previous similar messages [ 252.992833] LustreError: 18114:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 252.994748] LustreError: 18114:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 253.030695] Lustre: server umount lustre-MDT0000 complete [ 254.555155] LustreError: 14966:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141334 with bad export cookie 18234293640179555375 [ 254.559340] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 254.559475] LustreError: 14966:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 254.733884] Lustre: server umount lustre-MDT0001 complete [ 257.437636] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:55:37 (1761141337) [ 266.120326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 266.356937] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 266.389785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 268.273673] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 271.843164] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 272.260792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 274.257371] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 276.677399] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 278.336731] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 279.282636] Lustre: Failing over lustre-MDT0000 [ 279.354026] LustreError: 20792:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 279.357314] LustreError: 20792:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 279.391589] Lustre: server umount lustre-MDT0000 complete [ 282.082436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 282.083236] LustreError: 19444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 282.087302] Lustre: Skipped 2 previous similar messages [ 293.890720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 293.969430] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 296.023385] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 298.211162] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 298.217077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 298.228408] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 300.127921] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 303.586401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 303.586568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.587378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.599623] Lustre: Skipped 1 previous similar message [ 308.616127] LustreError: 21771:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 308.620036] LustreError: 21771:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 308.663359] Lustre: server umount lustre-MDT0000 complete [ 308.706926] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 308.714367] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 310.461451] LustreError: 19429:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141390 with bad export cookie 18234293640179556789 [ 310.466770] LustreError: 19429:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 1 previous similar message [ 310.468385] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 310.697550] Lustre: server umount lustre-MDT0001 complete [ 315.329269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 315.687306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 315.692196] Lustre: Skipped 2 previous similar messages [ 317.902709] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 322.470307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 324.626935] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 327.324489] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 329.146957] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 330.256066] Lustre: Failing over lustre-MDT0000 [ 330.373286] Lustre: server umount lustre-MDT0000 complete [ 331.235648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.239735] Lustre: Skipped 1 previous similar message [ 342.498625] LustreError: 22449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.505503] LustreError: 22449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 344.791572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 344.848276] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 346.961743] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 348.067791] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 348.075498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 348.078435] Lustre: Skipped 1 previous similar message [ 348.089207] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 351.122705] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 358.370127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 358.372156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.378248] Lustre: Skipped 2 previous similar messages [ 359.624760] LustreError: 24774:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 359.627455] LustreError: 24774:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 359.667300] Lustre: server umount lustre-MDT0000 complete [ 361.396607] LustreError: 22429:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141441 with bad export cookie 18234293640179557720 [ 361.401503] LustreError: 22429:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 361.402168] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 361.607801] Lustre: server umount lustre-MDT0001 complete [ 365.144806] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:57:24 (1761141444) [ 371.688764] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 375.654385] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 380.509349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 383.560547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 386.433188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 389.291446] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 393.978991] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 398.538889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 398.572967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.683441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 398.701423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 398.748520] Lustre: lustre-MDT0000: new disk, initializing [ 398.780667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 398.783841] Lustre: Skipped 2 previous similar messages [ 398.791341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 400.496198] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 405.573473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 405.604396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 405.648347] Lustre: 28438:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 405.787343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 407.508536] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 410.910377] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 412.338317] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 414.799431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 414.827465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 416.656075] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 416.696296] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 416.705500] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 417.333614] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 420.820594] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 422.952616] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 429.505080] LustreError: 31075:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 429.507783] LustreError: 31075:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 429.588925] Lustre: server umount lustre-OST0000 complete [ 431.585418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 431.590056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.595514] Lustre: Skipped 2 previous similar messages [ 431.597676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.318584] Lustre: server umount lustre-MDT0000 complete [ 441.648613] LustreError: 28429:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141521 with bad export cookie 18234293640179558119 [ 441.651790] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 441.655430] LustreError: 28429:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 452.184880] Lustre: server umount lustre-MDT0001 complete [ 456.769819] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 458.087450] Key type lgssc unregistered [ 458.253629] LNet: 32010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.257852] LNetError: 32010:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 458.266825] LNet: Removed LNI 192.168.201.109@tcp [ 458.675393] Key type .llcrypt unregistered [ 458.677245] Key type ._llcrypt unregistered [ 469.322328] Key type ._llcrypt registered [ 469.324570] Key type .llcrypt registered [ 469.373038] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 469.835470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 469.899095] alg: No test for adler32 (adler32-zlib) [ 470.768267] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 470.868076] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 472.456192] Key type lgssc registered [ 472.898936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 475.969871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 475.977632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 477.188092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.851633] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 482.513987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 482.668197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 484.436563] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 486.781764] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 488.303912] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 491.252960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 491.440074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.456787] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 494.045638] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 496.407742] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 508.653167] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:59:48 (1761141588) [ 509.577345] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 510.470483] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 513.294490] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:59:53 (1761141593) [ 516.996473] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:59:56 (1761141596) [ 517.476548] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 521.273644] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 521.276320] Lustre: Skipped 2 previous similar messages [ 522.070269] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:00:01 (1761141601) [ 525.938475] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 525.941459] Lustre: Skipped 4 previous similar messages [ 532.698952] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:00:12 (1761141612) [ 533.869794] Lustre: *** cfs_fail_loc=131b, val=2*** [ 535.378149] Lustre: *** cfs_fail_loc=131b, val=2*** [ 539.163480] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:00:18 (1761141618) [ 540.130697] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 540.135790] Lustre: Skipped 9 previous similar messages [ 546.357418] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:00:26 (1761141626) [ 552.931509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 552.933431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.934237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 552.944421] Lustre: Skipped 1 previous similar message [ 557.746575] LustreError: 39935:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 557.823912] Lustre: server umount lustre-OST0000 complete [ 559.585145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 559.587501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.592957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 559.596236] Lustre: Skipped 1 previous similar message [ 563.170618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.170803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 565.569609] LustreError: 40136:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 565.571651] LustreError: 40136:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 565.603509] Lustre: server umount lustre-MDT0000 complete [ 567.251370] LustreError: 40138:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141647 with bad export cookie 15906923444159290195 [ 567.254555] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 567.255460] LustreError: 40138:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 567.334620] LustreError: 40339:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 567.338770] LustreError: 40339:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 567.461188] Lustre: server umount lustre-MDT0001 complete [ 572.378731] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 573.858057] Key type lgssc unregistered [ 574.042709] LNet: 40870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.046415] LNetError: 40870:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 574.058532] LNet: Removed LNI 192.168.201.109@tcp [ 574.499238] Key type .llcrypt unregistered [ 574.501239] Key type ._llcrypt unregistered [ 582.436940] Key type ._llcrypt registered [ 582.438659] Key type .llcrypt registered [ 582.490887] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 590.314666] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 590.789475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 590.797879] alg: No test for adler32 (adler32-zlib) [ 591.662253] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 591.755178] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 593.336158] Key type lgssc registered [ 593.767915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 597.626511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 600.417740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 603.098966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 605.891852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 608.807497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 608.830875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 608.838684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.946066] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 609.963562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 610.009344] Lustre: lustre-MDT0000: new disk, initializing [ 610.051752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 610.062413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 611.792964] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 616.914308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 616.951032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 616.998448] Lustre: 43985:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 617.039322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 617.042618] Lustre: Skipped 1 previous similar message [ 617.095814] Lustre: lustre-MDT0001: new disk, initializing [ 617.145766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 617.165451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 617.170711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 619.014334] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 623.014936] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 624.747340] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 627.774165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 627.807611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 627.943546] Lustre: lustre-OST0000: new disk, initializing [ 627.946038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 627.999619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 630.531317] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 633.860097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 633.865036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 633.915405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 634.414480] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 636.279398] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 636.376776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 638.098713] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 638.210947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 644.067041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.067746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 644.077706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 644.079149] LustreError: Skipped 1 previous similar message [ 645.167205] LustreError: 46925:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 645.251043] Lustre: server umount lustre-OST0000 complete [ 648.161241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 648.164539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 648.170883] Lustre: Skipped 1 previous similar message [ 648.173275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 648.176349] Lustre: Skipped 1 previous similar message [ 649.185700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 649.186432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 653.252826] LustreError: 47127:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 653.257186] LustreError: 47127:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 653.291449] Lustre: server umount lustre-MDT0000 complete [ 654.906334] LustreError: 43978:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141734 with bad export cookie 2063047831333329196 [ 654.907860] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 654.912596] LustreError: 43978:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 655.004509] LustreError: 47329:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 655.007882] LustreError: 47329:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 655.123743] Lustre: server umount lustre-MDT0001 complete [ 659.443123] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:02:19 (1761141739) [ 662.479472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.710513] LustreError: 48048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 662.718448] LustreError: 48048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 662.745875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 664.450632] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 668.133097] LustreError: 48048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 668.379673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 668.537685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 670.367497] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 672.720663] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 674.325288] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 677.410404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 677.555948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.072543] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 682.451972] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 691.036919] LustreError: 50246:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 691.434760] Lustre: Modifying parameter general.timeout=1 in log params [ 692.556766] Lustre: Modifying parameter general.timeout=4 in log params [ 692.559295] Lustre: Skipped 2 previous similar messages [ 694.802733] Lustre: Modifying parameter general.timeout=10 in log params [ 694.805038] Lustre: Skipped 5 previous similar messages [ 699.023534] Lustre: Modifying parameter general.timeout=20 in log params [ 699.026648] Lustre: Skipped 9 previous similar messages [ 707.234171] Lustre: Modifying parameter general.timeout=40 in log params [ 707.236281] Lustre: Skipped 19 previous similar messages [ 723.288079] Lustre: Modifying parameter general.timeout=80 in log params [ 723.291699] Lustre: Skipped 39 previous similar messages [ 746.188655] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:03:46 (1761141826) [ 747.412303] Lustre: 57145:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 749.015976] Lustre: 57343:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 749.024481] Lustre: 57343:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 751.402122] Lustre: 57659:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 751.407345] Lustre: 57659:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 2 previous similar messages [ 755.642101] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 in log params [ 755.645456] Lustre: Skipped 64 previous similar messages [ 761.965584] Lustre: 59000:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 761.971658] LustreError: 59000:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 763.517759] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:04:03 (1761141843) [ 768.995094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 768.995989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 769.001969] Lustre: Skipped 2 previous similar messages [ 769.004607] Lustre: Skipped 2 previous similar messages [ 774.019110] LustreError: 59696:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 774.021591] LustreError: 59696:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 774.050796] Lustre: server umount lustre-MDT0000 complete [ 774.115100] LustreError: 48048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 774.123723] LustreError: 48048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 775.451570] LustreError: 48028:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141855 with bad export cookie 2063047831333330064 [ 775.455172] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 775.458534] LustreError: 48028:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 775.626994] Lustre: server umount lustre-MDT0001 complete [ 786.354800] LustreError: 60099:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 786.357517] LustreError: 60099:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 786.439554] Lustre: server umount lustre-OST0000 complete [ 789.777289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 791.347107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 792.905086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 794.397294] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 796.948830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.060373] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 797.192188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 798.774622] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 802.101761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 802.147450] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 802.159569] Lustre: 61547:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 803.845977] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 806.106318] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 807.613726] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 810.395612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 810.514442] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 812.991659] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 815.223712] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 822.754951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 822.755874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 822.759188] LustreError: Skipped 1 previous similar message [ 822.766448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 822.770641] Lustre: Skipped 2 previous similar messages [ 824.175189] LustreError: 64025:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 824.177643] LustreError: 64025:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 824.252359] Lustre: server umount lustre-OST0000 complete [ 827.875199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 832.994241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 832.997318] Lustre: Skipped 3 previous similar messages [ 840.160816] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 840.211899] LustreError: 64234:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 840.217097] LustreError: 64234:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 840.254565] Lustre: server umount lustre-MDT0000 complete [ 841.957505] LustreError: 63780:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141921 with bad export cookie 2063047831333332430 [ 841.961140] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 841.963156] LustreError: 63780:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 842.143631] Lustre: server umount lustre-MDT0001 complete [ 846.935430] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 848.240396] Key type lgssc unregistered [ 848.387331] LNet: 64969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.390239] LNetError: 64969:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.398565] LNet: Removed LNI 192.168.201.109@tcp [ 848.760747] Key type .llcrypt unregistered [ 848.761991] Key type ._llcrypt unregistered [ 859.431365] Key type ._llcrypt registered [ 859.432846] Key type .llcrypt registered [ 859.479709] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 859.884295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 859.936756] alg: No test for adler32 (adler32-zlib) [ 860.808925] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 860.917083] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 862.512176] Key type lgssc registered [ 862.989804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 866.446841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 866.460490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.718352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 869.565192] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 873.334525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 873.602102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 875.435213] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 877.831444] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 879.530280] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 882.678046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 882.945527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 885.537194] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 887.982975] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 897.528411] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 903.651483] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 903.652568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 903.658434] Lustre: Skipped 1 previous similar message [ 907.013845] LustreError: 69052:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 907.048255] Lustre: server umount lustre-MDT0000 complete [ 908.542320] LustreError: 66320:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141988 with bad export cookie 14675050442698458237 [ 908.546738] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 908.549145] LustreError: 66320:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 908.628180] LustreError: 69256:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 908.631955] LustreError: 69256:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 908.718926] Lustre: server umount lustre-MDT0001 complete [ 920.635216] LustreError: 69457:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 920.639388] LustreError: 69457:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 920.729762] Lustre: server umount lustre-OST0000 complete [ 924.872826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 926.624601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 928.308313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 929.948575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 933.078606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 933.203690] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 933.221374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 933.337961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.150671] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 939.240257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 939.287715] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 939.302643] Lustre: 70906:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 939.543155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 941.274194] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 943.745941] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 945.381953] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 948.312712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 948.397179] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 948.575808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.180956] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 953.608889] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 954.299405] Lustre: 73149:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 955.105834] Lustre: Modifying parameter general.jobid_var=disable in log params [ 956.901412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 956.902642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.905469] LustreError: Skipped 1 previous similar message [ 956.911584] Lustre: Skipped 1 previous similar message [ 956.914041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 956.918329] Lustre: Skipped 3 previous similar messages [ 958.771306] LustreError: 73398:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 958.774563] LustreError: 73398:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 958.864171] Lustre: server umount lustre-OST0000 complete [ 962.019816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.020695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 962.027499] Lustre: Skipped 1 previous similar message [ 962.613624] Lustre: server umount lustre-MDT0000 complete [ 964.264834] LustreError: 70897:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142044 with bad export cookie 14675050442698459056 [ 964.269150] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 964.269933] LustreError: 70897:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 964.347493] LustreError: 73807:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 964.351233] LustreError: 73807:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 964.444924] Lustre: server umount lustre-MDT0001 complete [ 968.918621] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 970.326162] Key type lgssc unregistered [ 970.484683] LNet: 74338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.488855] LNetError: 74338:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.498521] LNet: Removed LNI 192.168.201.109@tcp [ 970.880050] Key type .llcrypt unregistered [ 970.881539] Key type ._llcrypt unregistered [ 982.186234] Key type ._llcrypt registered [ 982.187925] Key type .llcrypt registered [ 982.234121] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 982.777811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 982.786588] alg: No test for adler32 (adler32-zlib) [ 983.676771] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 983.774131] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 985.368158] Key type lgssc registered [ 985.793463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 988.861378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 988.871363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.055516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.624287] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 995.154812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 995.413312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 997.024401] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 999.204309] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1000.642178] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1003.557263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1003.807942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.104056] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1008.306871] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1021.296212] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1021.990523] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1021.992903] Lustre: Skipped 1 previous similar message [ 1023.130605] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1023.133500] Lustre: Skipped 2 previous similar messages [ 1025.389295] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1025.392571] Lustre: Skipped 5 previous similar messages [ 1029.650645] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1029.654058] Lustre: Skipped 11 previous similar messages [ 1037.807877] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1037.810247] Lustre: Skipped 22 previous similar messages [ 1053.941738] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1053.945515] Lustre: Skipped 48 previous similar messages [ 1058.266229] Lustre: 83464:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1059.188595] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:08:59 (1761142139) [ 1059.767410] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1060.420356] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:09:00 (1761142140) [ 1062.368995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1062.371238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1062.376083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1066.467446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1066.470494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1066.471043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1067.567160] LustreError: 83758:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1067.637660] Lustre: server umount lustre-OST0000 complete [ 1071.585547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1071.586270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1071.589207] Lustre: Skipped 1 previous similar message [ 1071.592143] Lustre: Skipped 1 previous similar message [ 1076.705351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1076.707635] Lustre: Skipped 1 previous similar message [ 1081.825389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1081.827164] Lustre: Skipped 1 previous similar message [ 1083.360097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1083.396722] LustreError: 83959:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1083.398719] LustreError: 83959:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1083.427579] Lustre: server umount lustre-MDT0000 complete [ 1084.730603] LustreError: 75688:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142164 with bad export cookie 14418372889771891886 [ 1084.733953] LustreError: 75688:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1084.734207] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1084.809305] LustreError: 84163:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1084.810919] LustreError: 84163:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1084.905069] Lustre: server umount lustre-MDT0001 complete [ 1088.960588] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1090.203258] Key type lgssc unregistered [ 1090.338342] LNet: 84694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.341116] LNetError: 84694:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1090.348540] LNet: Removed LNI 192.168.201.109@tcp [ 1090.698815] Key type .llcrypt unregistered [ 1090.700078] Key type ._llcrypt unregistered [ 1092.421142] Key type ._llcrypt registered [ 1092.422217] Key type .llcrypt registered [ 1092.470896] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1093.979135] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules [ 1094.388282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1094.456084] alg: No test for adler32 (adler32-zlib) [ 1095.334469] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 1095.433808] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1097.016163] Key type lgssc registered [ 1097.497706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1097.943432] LustreError: 85810:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1105.785886] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1109.391807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1119.054551] LustreError: 85810:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1119.067818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.272031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.696898] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1124.558855] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:10:04 (1761142204) [ 1126.195303] LustreError: 86848:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1126.199575] LustreError: 86848:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1126.203620] LustreError: 86848:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 1126.264445] LustreError: 87620:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1126.356708] Lustre: server umount lustre-MDT0000 complete [ 1131.236145] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1132.482979] Key type lgssc unregistered [ 1132.640632] LNet: 88305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1132.645335] LNetError: 88305:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1132.652368] LNet: Removed LNI 192.168.201.109@tcp [ 1132.994893] Key type .llcrypt unregistered [ 1132.996327] Key type ._llcrypt unregistered [ 1140.719705] Key type ._llcrypt registered [ 1140.721154] Key type .llcrypt registered [ 1140.766941] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1141.267236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1141.285874] alg: No test for adler32 (adler32-zlib) [ 1142.134070] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 1142.210344] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1143.792382] Key type lgssc registered [ 1144.231483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1146.890648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1146.898427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.093704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1149.431952] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1152.260607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1152.483076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1153.933914] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1155.948763] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1157.315352] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1159.826770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1160.053643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.082518] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1164.135614] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1196.512637] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:11:16 (1761142276) [ 1197.172415] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1197.856612] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:11:17 (1761142277) [ 1203.170195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.171369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1203.175817] Lustre: Skipped 2 previous similar messages [ 1203.178898] Lustre: Skipped 2 previous similar messages [ 1205.055789] LustreError: 92413:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1205.087581] Lustre: server umount lustre-MDT0000 complete [ 1206.389468] LustreError: 89463:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142286 with bad export cookie 14728676566011377569 [ 1206.393428] LustreError: 89463:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1206.394928] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1206.463454] LustreError: 92616:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1206.465903] LustreError: 92616:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1206.546173] Lustre: server umount lustre-MDT0001 complete [ 1218.164026] LustreError: 92817:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1218.165959] LustreError: 92817:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1218.327484] Lustre: server umount lustre-OST0000 complete [ 1222.547729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.706721] LustreError: 93446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.712040] LustreError: 93446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1222.764395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1224.092495] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1227.148825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1227.440708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1228.986092] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1231.003360] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1232.468657] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1234.740368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1237.013846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1237.039045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1237.107855] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1237.111018] LustreError: 93435:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1237.121849] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1237.124714] LustreError: 95307:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1237.127621] LustreError: 95307:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1237.129946] LustreError: 95307:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1237.177140] Lustre: server umount lustre-OST0000 complete [ 1237.178542] LustreError: 95307:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1240.124858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1240.191826] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1240.194901] LustreError: 93436:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1240.205815] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1240.209258] LustreError: 95771:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1240.212194] LustreError: 95771:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1240.214126] LustreError: 95771:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1240.262858] Lustre: server umount lustre-OST0000 complete [ 1240.264449] LustreError: 95771:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1241.255259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1243.651522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1243.719913] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1243.723528] Lustre: Found index 0 for lustre-OST0000, updating log [ 1243.726549] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1243.767981] Lustre: lustre-OST0000: new disk, initializing [ 1243.771735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1243.887387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.028691] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1247.324078] Lustre: Failing over lustre-OST0000 [ 1247.343238] LustreError: 96845:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1247.345781] LustreError: 96845:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1247.427534] Lustre: server umount lustre-OST0000 complete [ 1252.835720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1252.835872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.837083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1252.846965] Lustre: Skipped 1 previous similar message [ 1257.953116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.956994] Lustre: Skipped 2 previous similar messages [ 1263.072112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1263.073260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.078482] Lustre: Skipped 1 previous similar message [ 1263.110258] LustreError: 97053:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1263.114060] LustreError: 97053:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1263.149246] Lustre: server umount lustre-MDT0000 complete [ 1264.541301] LustreError: 93474:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142344 with bad export cookie 14728676566011380299 [ 1264.543588] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1264.545516] LustreError: 93474:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1264.705951] Lustre: server umount lustre-MDT0001 complete [ 1267.988084] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:12:27 (1761142347) [ 1273.578299] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1277.421968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.564604] LustreError: 98622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.595164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.104927] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1282.150532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1283.731372] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1284.784605] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1287.178068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1289.427886] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1292.494360] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1292.525971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1292.557490] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1292.606872] Lustre: lustre-OST0001: new disk, initializing [ 1292.608962] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1294.406049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1294.410973] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1294.444267] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1294.724588] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1298.012680] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1303.842145] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1311.238648] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 1313.249045] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1313.251659] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1315.299194] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1315.303673] Lustre: Skipped 1 previous similar message [ 1316.416129] LustreError: 101865:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1316.418209] LustreError: 101865:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1316.439308] Lustre: server umount lustre-MDT0001 complete [ 1317.414856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1319.776830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1319.813983] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1319.816179] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1319.832510] Lustre: 98611:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1320.114423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1320.117871] Lustre: Skipped 3 previous similar messages [ 1321.382799] LustreError: 102599:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1321.385099] LustreError: 102599:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1321.387511] LustreError: 98613:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1321.416835] LustreError: 102613:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.109@tcp: cfg command failed: rc = -17 [ 1321.421380] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1321.421380] [ 1321.500579] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1325.147146] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 10:13:24 (1761142404) [ 1325.548959] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1331.664711] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1337.966525] Lustre: 104469:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1346.540764] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1347.904108] Lustre: 105601:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1401.824725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1401.828190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.834174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1401.836391] Lustre: Skipped 5 previous similar messages [ 1407.623725] LustreError: 106056:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1407.625760] LustreError: 106056:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1407.649854] Lustre: server umount lustre-MDT0000 complete [ 1409.017849] LustreError: 100891:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142488 with bad export cookie 14728676566011381209 [ 1409.020448] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1409.022516] LustreError: 100891:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1409.180171] Lustre: server umount lustre-MDT0001 complete [ 1420.812547] Lustre: server umount lustre-OST0000 complete [ 1432.726583] Lustre: server umount lustre-OST0001 complete [ 1434.770735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1438.074444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1438.449302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1441.382877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1444.884213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1445.323147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1452.489591] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1456.494984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.500659] Lustre: lustre-MDT0000: reset Object Index mappings [ 1456.674568] LustreError: 109533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.707625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.156075] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1461.579965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1461.587113] Lustre: lustre-MDT0001: reset Object Index mappings [ 1463.539647] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1464.721070] Lustre: 110647:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1467.260444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1469.759371] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1471.654465] LustreError: 111005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1471.659693] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 1472.961856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1474.992887] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1475.319365] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1482.153381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 1482.155261] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1489.098403] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1490.435615] Lustre: 112526:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1512.198299] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:16:32 (1761142592) [ 1543.652639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1543.653947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1543.656980] Lustre: Skipped 5 previous similar messages [ 1543.658594] Lustre: Skipped 9 previous similar messages [ 1543.660109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1546.950543] LustreError: 112873:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1546.953130] LustreError: 112873:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1546.983619] Lustre: server umount lustre-MDT0000 complete [ 1548.236237] LustreError: 109512:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142628 with bad export cookie 14728676566011427976 [ 1548.239744] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1548.241019] LustreError: 109512:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1548.769328] LustreError: 109527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.774943] LustreError: 109527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1554.513370] Lustre: server umount lustre-MDT0001 complete [ 1570.272147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1570.318331] Lustre: server umount lustre-OST0000 complete [ 1577.719460] Lustre: server umount lustre-OST0001 complete [ 1579.805349] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 1583.319801] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1587.005852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1589.493210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1591.630950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1593.909814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1596.306572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1596.329416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.416249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1596.428807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1596.469766] Lustre: lustre-MDT0000: new disk, initializing [ 1596.495074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1596.497454] Lustre: Skipped 3 previous similar messages [ 1596.503750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1597.869657] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1600.981216] Lustre: server umount lustre-MDT0000 complete [ 1601.932205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1604.204447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1604.228283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.328129] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1604.332671] Lustre: 116755:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1604.337579] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1605.744976] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1607.553310] Lustre: server umount lustre-MDT0000 complete [ 1610.490225] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:18:10 (1761142690) [ 1611.048145] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1611.661382] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:18:11 (1761142691) [ 1617.840765] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1621.551066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1623.053603] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1625.912918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1625.932708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1625.964088] Lustre: 118774:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1625.989223] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1625.991238] Lustre: Skipped 1 previous similar message [ 1626.029507] Lustre: lustre-MDT0001: new disk, initializing [ 1626.075584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1626.079712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1627.450627] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1629.816506] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1632.104285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1632.125700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1632.221485] Lustre: lustre-OST0000: new disk, initializing [ 1632.223270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1633.467503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1633.470721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1633.503219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1634.255118] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1638.428062] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1638.457809] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1640.520499] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1640.520511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1640.527102] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1640.560282] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1645.146786] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1646.445794] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1805.462317] Lustre: 127538:0:(osd_internal.h:1337:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1805.464881] Lustre: 127538:0:(osd_handler.c:2080:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1805.466584] Lustre: 127538:0:(osd_handler.c:2087:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1805.468851] Lustre: 127538:0:(osd_handler.c:2097:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1805.472296] Lustre: 127538:0:(osd_handler.c:2104:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1805.474355] Lustre: 127538:0:(osd_handler.c:2111:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1876.149474] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:22:35 (1761142955) [ 1889.798060] LustreError: 127741:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1889.799974] LustreError: 127741:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 1889.861330] Lustre: server umount lustre-MDT0000 complete [ 1891.112963] LustreError: 118764:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142970 with bad export cookie 14728676566011482534 [ 1891.118697] LustreError: 118764:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1891.118734] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1891.810425] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.811255] LustreError: 118779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1891.811476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1891.811481] Lustre: Skipped 5 previous similar messages [ 1891.813688] Lustre: Skipped 5 previous similar messages [ 1891.821911] LustreError: 118779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1897.381713] Lustre: server umount lustre-MDT0001 complete [ 1913.312165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1913.352166] Lustre: server umount lustre-OST0000 complete [ 1920.759440] Lustre: server umount lustre-OST0001 complete [ 1923.011870] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 1926.445555] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1930.401448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1932.737301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1934.854420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1937.059101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1939.496462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1939.524058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1939.632557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1939.648947] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1939.650508] Lustre: Skipped 1 previous similar message [ 1939.694163] Lustre: lustre-MDT0000: new disk, initializing [ 1939.695378] Lustre: Skipped 1 previous similar message [ 1939.723838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.727249] Lustre: Skipped 5 previous similar messages [ 1939.735205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1941.180859] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1945.495169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1945.522912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1945.558632] Lustre: 130527:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1945.664150] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1947.027090] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1950.128303] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1951.345220] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1953.613616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1953.642089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1955.650884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1955.699149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1955.882273] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1959.073878] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1962.310567] Lustre: lustre-MDD0000: changelog on [ 1963.224928] Lustre: lustre-MDD0001: changelog on [ 2328.612583] Lustre: lustre-MDD0001: changelog off [ 2329.375530] Lustre: 134238:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2329.801644] Lustre: lustre-MDD0000: changelog off [ 2334.689568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2334.691552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.694409] Lustre: Skipped 2 previous similar messages [ 2334.694667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2334.698968] Lustre: Skipped 4 previous similar messages [ 2337.970104] LustreError: 139753:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2337.973259] LustreError: 139753:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2338.056146] Lustre: server umount lustre-OST0000 complete [ 2339.809910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2339.812432] LustreError: Skipped 1 previous similar message [ 2345.561858] Lustre: server umount lustre-MDT0000 complete [ 2347.326924] LustreError: 131307:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143427 with bad export cookie 14728676566012250700 [ 2347.334345] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2347.336037] LustreError: 131307:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2347.522249] Lustre: server umount lustre-MDT0001 complete [ 2351.496228] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2352.556135] Key type lgssc unregistered [ 2352.695425] LNet: 140686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.697858] LNetError: 140686:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.704336] LNet: Removed LNI 192.168.201.109@tcp [ 2353.047228] Key type .llcrypt unregistered [ 2353.050884] Key type ._llcrypt unregistered [ 2361.233702] Key type ._llcrypt registered [ 2361.235400] Key type .llcrypt registered [ 2361.276545] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 2367.533332] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2367.910952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2367.979973] alg: No test for adler32 (adler32-zlib) [ 2368.839927] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 2368.928307] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2370.512168] Key type lgssc registered [ 2370.898666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2373.992206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2376.324710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2378.804250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2381.821441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2384.303840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2384.330462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2384.336274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.421609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2385.434918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2385.474822] Lustre: lustre-MDT0000: new disk, initializing [ 2385.501435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2385.508222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2386.779367] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2391.284978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2391.319932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2391.358470] Lustre: 143835:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2391.393351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2391.395242] Lustre: Skipped 1 previous similar message [ 2391.430592] Lustre: lustre-MDT0001: new disk, initializing [ 2391.465778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2391.476579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2391.479520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2392.896149] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2396.445724] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2397.736473] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2400.359352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2400.387611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2400.490927] Lustre: lustre-OST0000: new disk, initializing [ 2400.493396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2400.522828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.440623] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2405.537253] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2411.011693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2411.014432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2411.044985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2417.307700] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_module obdecho/obdecho [ 2417.905244] LustreError: 146480:0:(echo_client.c:669:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2417.908388] LustreError: 146480:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 2418.505755] LustreError: 146577:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2422.240912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2422.244722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.248588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2426.338056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2426.340257] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.343797] Lustre: Skipped 1 previous similar message [ 2427.903136] LustreError: 146917:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2427.936463] Lustre: server umount lustre-MDT0000 complete [ 2429.185385] LustreError: 143828:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143508 with bad export cookie 2921364148165697121 [ 2429.189651] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2429.192043] LustreError: 143828:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2429.265569] LustreError: 147118:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2429.268837] LustreError: 147118:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2429.363424] Lustre: server umount lustre-MDT0001 complete [ 2440.881423] LustreError: 147318:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2440.883428] LustreError: 147318:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2440.945179] Lustre: server umount lustre-OST0000 complete [ 2447.997810] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 2451.892556] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2455.528345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2457.912980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2460.038865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2462.339565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2464.716089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2464.740496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2464.822109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2464.834809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2464.873778] Lustre: lustre-MDT0000: new disk, initializing [ 2464.900750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2464.907683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2466.271913] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2470.583182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2470.611483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2470.643523] Lustre: 150259:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2470.674513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2470.676129] Lustre: Skipped 1 previous similar message [ 2470.714172] Lustre: lustre-MDT0001: new disk, initializing [ 2470.752647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2470.765091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2470.769701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2472.167493] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2475.293464] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2476.526741] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2478.913689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2478.946447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2479.067467] Lustre: lustre-OST0000: new disk, initializing [ 2479.070942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2479.106191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.042917] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2483.008513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2483.013419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2483.070160] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2484.235778] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2485.641565] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2485.723618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2487.213376] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2487.303612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2488.290043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2488.292415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.295178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2488.296298] Lustre: Skipped 1 previous similar message [ 2488.297934] Lustre: Skipped 2 previous similar messages [ 2489.966088] LustreError: 153197:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2489.967779] LustreError: 153197:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2490.037061] Lustre: server umount lustre-OST0000 complete [ 2491.360975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2491.363752] LustreError: Skipped 1 previous similar message [ 2491.365968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2491.372655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2497.407582] LustreError: 153399:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2497.409723] LustreError: 153399:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2497.438581] Lustre: server umount lustre-MDT0000 complete [ 2498.754387] LustreError: 151048:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143578 with bad export cookie 2921364148165698066 [ 2498.759219] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2498.926352] Lustre: server umount lustre-MDT0001 complete [ 2500.207159] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:33:00 (1761143580) [ 2500.800526] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2501.430340] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:33:01 (1761143581) [ 2504.078295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2504.231034] LustreError: 154270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.237363] LustreError: 154270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2504.255338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2505.635817] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2508.579637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2510.071994] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2511.828108] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2512.985728] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2515.239312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2517.329753] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2519.225832] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2534.369267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2534.371379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.374222] Lustre: Skipped 1 previous similar message [ 2534.376478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2534.379034] Lustre: Skipped 2 previous similar messages [ 2537.736443] LustreError: 156538:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2537.739652] LustreError: 156538:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2537.776928] Lustre: server umount lustre-MDT0000 complete [ 2539.062569] LustreError: 154250:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143618 with bad export cookie 2921364148165698934 [ 2539.066749] LustreError: 154250:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2539.069150] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2540.001862] LustreError: 155886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.120892] LustreError: 155886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.121221] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2545.125529] LustreError: 155886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2545.127489] Lustre: Skipped 5 previous similar messages [ 2545.296767] Lustre: server umount lustre-MDT0001 complete [ 2547.500178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2551.992735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2552.077893] LustreError: 157798:0:(llog.c:1616:llog_backup()) MGC192.168.201.109@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2552.080669] Lustre: 157798:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.109@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2568.673069] LustreError: 157806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.677376] LustreError: 157806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2573.792959] LustreError: 157806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.797955] LustreError: 157806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2584.033565] LustreError: 157823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.037389] LustreError: 157823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2601.470474] LustreError: 157823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.475492] LustreError: 157823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 2612.832404] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2612.834088] Lustre: 157798:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.109@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2633.697591] LustreError: 157806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.701686] LustreError: 157806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 2640.352390] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2640.354462] Lustre: 154261:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.109@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2653.672543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2653.674898] Lustre: Skipped 2 previous similar messages [ 2654.890117] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2657.870190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.793060] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2658.798947] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x288ac536f6fa7eac [ 2660.170880] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2661.818495] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2662.404527] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2663.704352] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2664.425408] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2664.427077] Lustre: Skipped 1 previous similar message [ 2667.362455] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2672.596696] LustreError: 159283:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2672.599498] LustreError: 159283:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2672.638807] Lustre: server umount lustre-MDT0000 complete [ 2673.988337] LustreError: 154251:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143753 with bad export cookie 2921364148165705388 [ 2673.992925] LustreError: 154251:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2673.993125] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2674.130910] Lustre: server umount lustre-MDT0001 complete [ 2696.077478] Lustre: server umount lustre-OST0000 complete [ 2703.790616] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 2707.650968] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2711.635054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2713.797426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2715.807162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2717.931721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2720.285422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2720.305832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.389491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2720.403265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2720.446943] Lustre: lustre-MDT0000: new disk, initializing [ 2720.472625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2720.474387] Lustre: Skipped 1 previous similar message [ 2720.479445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2721.779798] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2726.344467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2726.369979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2726.409735] Lustre: 162627:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2726.450176] LustreError: 162639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.456866] LustreError: 162639:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 2726.537173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2727.826224] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2731.495972] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2732.832105] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2735.192991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2735.229811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2735.727599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2735.782377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2737.517164] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2740.717023] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2742.241967] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2742.316621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2743.642920] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2743.728751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2746.344757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.344965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2746.356160] Lustre: Skipped 3 previous similar messages [ 2746.358674] LustreError: Skipped 1 previous similar message [ 2746.359261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2746.365620] Lustre: Skipped 2 previous similar messages [ 2750.447125] LustreError: 165565:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2750.449587] LustreError: 165565:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2750.538799] Lustre: server umount lustre-OST0000 complete [ 2766.304088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2766.367948] Lustre: server umount lustre-MDT0000 complete [ 2767.976199] LustreError: 162618:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143847 with bad export cookie 2921364148165722160 [ 2767.981572] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2767.984622] LustreError: 162618:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2768.192134] Lustre: server umount lustre-MDT0001 complete [ 2772.333258] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:37:32 (1761143852) [ 2775.468173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2777.637240] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2781.227429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2782.714870] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2784.466409] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2785.635217] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2788.116386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2790.245714] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2792.149610] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2806.651872] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:38:06 (1761143886) [ 2810.852523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2810.855948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.856945] LustreError: Skipped 1 previous similar message [ 2810.860547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2810.860556] Lustre: Skipped 5 previous similar messages [ 2810.864397] Lustre: Skipped 3 previous similar messages [ 2814.064145] Lustre: server umount lustre-OST0000 complete [ 2821.822691] LustreError: 169305:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2821.825684] LustreError: 169305:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 2821.849582] Lustre: server umount lustre-MDT0000 complete [ 2823.128350] LustreError: 167360:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143902 with bad export cookie 2921364148165723021 [ 2823.133448] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2823.291160] Lustre: server umount lustre-MDT0001 complete [ 2827.160538] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2828.378912] Key type lgssc unregistered [ 2828.517371] LNet: 170037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2828.520216] LNetError: 170037:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2828.529584] LNet: Removed LNI 192.168.201.109@tcp [ 2828.867467] Key type .llcrypt unregistered [ 2828.868641] Key type ._llcrypt unregistered [ 2836.598277] Key type ._llcrypt registered [ 2836.599606] Key type .llcrypt registered [ 2836.636590] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2837.057045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2837.063751] alg: No test for adler32 (adler32-zlib) [ 2837.912821] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 2837.996906] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2839.576158] Key type lgssc registered [ 2839.930544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2842.566947] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2842.574698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2858.016524] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2863.136160] LustreError: 171236:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 2868.256432] LustreError: 171236:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2868.260060] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2868.262290] LustreError: MGC192.168.201.109@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2868.265384] LustreError: 171236:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2868.267835] LustreError: 171236:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 2868.271342] LustreError: 171236:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2868.273671] LustreError: 171236:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2868.342001] Lustre: server umount lustre-OST0000 complete [ 2868.344882] LustreError: 171236:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2871.424293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2872.592218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.982239] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2876.936942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2877.056594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2878.452701] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2880.268411] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2881.424586] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2883.903193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2883.992292] LustreError: 173325:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2883.995175] Lustre: 173325:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.201.109@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2884.056755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2886.054540] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2887.948231] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2892.262737] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.264416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2892.269609] Lustre: Skipped 1 previous similar message [ 2892.276949] Lustre: Skipped 2 previous similar messages [ 2895.038189] LustreError: 173979:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2895.039781] LustreError: 173979:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2895.065848] Lustre: server umount lustre-MDT0000 complete [ 2896.252282] LustreError: 171716:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143976 with bad export cookie 5087165792684011532 [ 2896.254885] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2896.260542] LustreError: 171716:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2896.327121] LustreError: 174181:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2896.328685] LustreError: 174181:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2896.405443] Lustre: server umount lustre-MDT0001 complete [ 2908.085246] LustreError: 174381:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2908.088295] LustreError: 174381:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2908.159391] Lustre: server umount lustre-OST0000 complete [ 2911.287353] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 2914.711440] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2918.608095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2921.129995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2923.208592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2925.669988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2927.991848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2928.008653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2928.095933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2928.110424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2928.146870] Lustre: lustre-MDT0000: new disk, initializing [ 2928.169990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2928.175815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2929.521430] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2934.416325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2934.445305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2934.487303] Lustre: 176715:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2934.530934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2934.535080] Lustre: Skipped 1 previous similar message [ 2934.580596] Lustre: lustre-MDT0001: new disk, initializing [ 2934.622829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2934.635677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2934.638872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2936.202974] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2939.583506] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2940.861122] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2943.336383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2943.366946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2943.490046] Lustre: lustre-OST0000: new disk, initializing [ 2943.492399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2943.528477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2944.385940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2944.389226] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2944.433187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2945.739813] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2949.300641] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2950.814802] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2950.886401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2952.268392] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2952.346610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2954.725790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2954.727089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.731289] LustreError: Skipped 1 previous similar message [ 2954.738571] Lustre: Skipped 1 previous similar message [ 2954.740088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2954.749470] Lustre: Skipped 1 previous similar message [ 2959.214230] LustreError: 179653:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2959.216447] LustreError: 179653:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2959.285674] Lustre: server umount lustre-OST0000 complete [ 2964.962609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.963735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.965618] Lustre: Skipped 2 previous similar messages [ 2964.970968] Lustre: Skipped 1 previous similar message [ 2970.081761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2970.084777] Lustre: Skipped 1 previous similar message [ 2975.200171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2975.201884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2975.204371] Lustre: Skipped 1 previous similar message [ 2975.234484] LustreError: 179856:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2975.236327] LustreError: 179856:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2975.260928] Lustre: server umount lustre-MDT0000 complete [ 2976.506200] LustreError: 176705:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144056 with bad export cookie 5087165792684012225 [ 2976.508633] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2976.512258] LustreError: 176705:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2976.675713] Lustre: server umount lustre-MDT0001 complete [ 2979.592244] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:40:59 (1761144059) [ 2985.538195] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2986.663862] Key type lgssc unregistered [ 2986.801327] LNet: 181244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2986.803680] LNetError: 181244:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2986.811278] LNet: Removed LNI 192.168.201.109@tcp [ 2987.127826] Key type .llcrypt unregistered [ 2987.128951] Key type ._llcrypt unregistered [ 2994.367506] Key type ._llcrypt registered [ 2994.369796] Key type .llcrypt registered [ 2994.418825] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2994.905382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2994.914574] alg: No test for adler32 (adler32-zlib) [ 2995.790405] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 2995.887168] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2997.488163] Key type lgssc registered [ 2997.837789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3000.540806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3000.548256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3001.758248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.199273] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3006.095261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3006.240809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3007.581590] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3009.414819] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3010.642913] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3013.197579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3013.286075] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3013.287532] LustreError: 184058:0:(llog.c:1657:llog_backup()) MGC192.168.201.109@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3013.290296] Lustre: 184058:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.201.109@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3013.334501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.349987] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3017.233261] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3017.955670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.958629] Lustre: Skipped 1 previous similar message [ 3017.959179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3017.961259] Lustre: Skipped 2 previous similar messages [ 3023.329666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3023.331364] Lustre: Skipped 2 previous similar messages [ 3024.074664] LustreError: 184706:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3024.103092] Lustre: server umount lustre-MDT0000 complete [ 3025.329319] LustreError: 182401:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144105 with bad export cookie 8661309805344355183 [ 3025.332448] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3025.333503] LustreError: 182401:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3025.393107] LustreError: 184912:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3025.396159] LustreError: 184912:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3025.474548] Lustre: server umount lustre-MDT0001 complete [ 3036.913024] LustreError: 185112:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3036.914914] LustreError: 185112:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3036.985507] Lustre: server umount lustre-OST0000 complete [ 3039.989874] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 3043.508919] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3047.385880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3049.828575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3052.134046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3054.498395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3057.030385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3057.061271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.178153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3057.189412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3057.233795] Lustre: lustre-MDT0000: new disk, initializing [ 3057.261344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3057.271166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3058.714720] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3063.017183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3063.054160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3063.090166] Lustre: 187448:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3063.111609] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3063.112996] Lustre: Skipped 1 previous similar message [ 3063.115458] LustreError: 188213:0:(llog.c:1657:llog_backup()) MGC192.168.201.109@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3063.118046] LustreError: 188213:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 3063.119764] Lustre: 188213:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.109@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3063.127513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3063.129339] Lustre: Skipped 1 previous similar message [ 3063.167483] Lustre: lustre-MDT0001: new disk, initializing [ 3063.201768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3063.213374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3063.216296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3064.551684] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3067.663683] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3068.886226] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3071.138963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3071.163150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3071.273730] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3071.275069] Lustre: Skipped 1 previous similar message [ 3071.276190] LustreError: 189169:0:(llog.c:1657:llog_backup()) MGC192.168.201.109@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3071.278629] LustreError: 189169:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 3071.280433] Lustre: 189169:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.109@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3071.283925] Lustre: 189169:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3071.295124] Lustre: lustre-OST0000: new disk, initializing [ 3071.298416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3071.333317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3073.243558] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3076.311431] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3077.720240] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3078.139805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3078.144708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3078.177601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3078.819169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3080.130358] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3080.215442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3083.237671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3083.238643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3083.241115] Lustre: Skipped 1 previous similar message [ 3083.243101] LustreError: Skipped 1 previous similar message [ 3083.243596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3083.247211] Lustre: Skipped 1 previous similar message [ 3086.959244] LustreError: 190392:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3086.963057] LustreError: 190392:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3087.035935] Lustre: server umount lustre-OST0000 complete [ 3088.353334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.357195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3088.358966] Lustre: Skipped 1 previous similar message [ 3093.473176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3093.475129] Lustre: Skipped 1 previous similar message [ 3094.401125] LustreError: 190593:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3094.403600] LustreError: 190593:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3094.433884] Lustre: server umount lustre-MDT0000 complete [ 3095.702758] LustreError: 190595:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144175 with bad export cookie 8661309805344355869 [ 3095.706958] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3095.708145] LustreError: 190595:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3095.851838] Lustre: server umount lustre-MDT0001 complete [ 3098.835652] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:42:58 (1761144178) [ 3104.228772] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3107.819189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.970050] LustreError: 192160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.975726] LustreError: 192160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3107.985995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3109.392567] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3112.655153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3114.003914] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3114.995256] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3117.280732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3119.267703] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3122.217558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3122.240067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3122.300813] Lustre: lustre-OST0001: new disk, initializing [ 3122.302819] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3124.277576] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3125.512962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3125.517870] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3125.553707] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3129.785543] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3135.463338] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3141.138457] Lustre: DEBUG MARKER: ADD OST3 [ 3142.739483] loop: module loaded [ 3142.758664] loop0: detected capacity change from 0 to 204800000 [ 3142.819161] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3142.823795] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3142.827734] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3142.886714] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3144.999666] loop0: detected capacity change from 0 to 204800000 [ 3146.993858] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3147.024852] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3147.101463] Lustre: lustre-OST0003: new disk, initializing [ 3147.103462] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3147.155512] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3147.157974] Lustre: Skipped 3 previous similar messages [ 3147.883791] Lustre: DEBUG MARKER: STOP OST3 [ 3148.388278] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3148.428170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3148.430653] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3148.440808] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3148.631026] Lustre: Failing over lustre-OST0003 [ 3148.656115] LustreError: 196481:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3148.658034] LustreError: 196481:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3148.680852] Lustre: server umount lustre-OST0003 complete [ 3149.391042] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3150.432072] LustreError: 196429:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3153.249480] Lustre: DEBUG MARKER: START OST3 again [ 3153.415801] LustreError: 193611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3153.894859] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3153.897413] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.902419] Lustre: Skipped 2 previous similar messages [ 3155.665412] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3155.754843] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3157.422617] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3157.650630] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3157.651150] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3157.716706] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3171.250942] Lustre: Failing over lustre-OST0003 [ 3171.274137] LustreError: 198451:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3171.275938] LustreError: 198451:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3171.292796] Lustre: server umount lustre-OST0003 complete [ 3171.305174] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.310929] LustreError: 193611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3171.314853] LustreError: 193611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3176.422094] LustreError: 193797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3176.431914] LustreError: 193797:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3181.542634] LustreError: 193611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3181.549483] LustreError: 193611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3184.608965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3184.613192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.617755] Lustre: Skipped 1 previous similar message [ 3184.620350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3190.064853] Lustre: server umount lustre-MDT0000 complete [ 3191.518303] LustreError: 192142:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144271 with bad export cookie 8661309805344356737 [ 3191.525249] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3191.777275] LustreError: 193796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3191.785925] LustreError: 193796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3197.950300] Lustre: server umount lustre-MDT0001 complete [ 3199.451120] Lustre: server umount lustre-OST0000 complete [ 3200.941081] Lustre: server umount lustre-OST0001 complete [ 3203.265594] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 3206.226176] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3210.182531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3212.797593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3214.923513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3217.531505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3219.992590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3220.025968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.130825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3220.146073] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3220.183367] Lustre: lustre-MDT0000: new disk, initializing [ 3220.211392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3220.214750] Lustre: Skipped 1 previous similar message [ 3220.222495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3221.700421] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3226.150281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3226.185536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3226.221258] Lustre: 201505:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3226.345593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3227.637528] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3230.811036] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3232.017546] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3234.350761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3234.376778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3236.284909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3236.287142] Lustre: Skipped 1 previous similar message [ 3236.288812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3236.329233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3236.470856] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3239.609166] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3241.102540] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3241.183226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3242.600576] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3242.677727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3246.562906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3246.564063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.565010] LustreError: Skipped 1 previous similar message [ 3246.568513] Lustre: Skipped 5 previous similar messages [ 3246.569198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3246.572732] Lustre: Skipped 9 previous similar messages [ 3249.453100] LustreError: 204439:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3249.455324] LustreError: 204439:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 3249.527904] Lustre: server umount lustre-OST0000 complete [ 3256.985173] Lustre: server umount lustre-MDT0000 complete [ 3258.224615] LustreError: 204642:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144338 with bad export cookie 8661309805344359019 [ 3258.228196] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3258.228366] LustreError: 204642:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3258.386369] Lustre: server umount lustre-MDT0001 complete [ 3259.550599] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:45:39 (1761144339) [ 3264.861147] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 3268.310841] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3271.984179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3274.413682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3276.500494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3278.676841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3281.111518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3281.139831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3281.233215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3281.246091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3281.248377] Lustre: Skipped 3 previous similar messages [ 3281.289668] Lustre: lustre-MDT0000: new disk, initializing [ 3281.291919] Lustre: Skipped 2 previous similar messages [ 3281.323442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3282.768616] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3287.057307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3287.080025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3287.113737] Lustre: 207682:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3287.221908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3288.596523] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3291.771514] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3293.197597] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3295.783540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3295.806125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3297.441784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3298.001199] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3301.671068] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3303.369211] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3303.447622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3304.874132] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3304.950604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3308.008196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.010806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3308.016837] Lustre: Skipped 3 previous similar messages [ 3308.019595] LustreError: Skipped 1 previous similar message [ 3308.020200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3308.025854] Lustre: Skipped 5 previous similar messages [ 3311.753597] Lustre: server umount lustre-OST0000 complete [ 3319.485516] LustreError: 210823:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3319.487204] LustreError: 210823:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3319.517288] Lustre: server umount lustre-MDT0000 complete [ 3320.740762] LustreError: 208464:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144400 with bad export cookie 8661309805344359901 [ 3320.744579] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3320.745794] LustreError: 208464:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3320.901961] Lustre: server umount lustre-MDT0001 complete [ 3323.666225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.840407] LustreError: 211503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3323.847931] LustreError: 211503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3325.124884] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3328.056677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3329.622290] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3331.409315] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3332.804134] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3335.458738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3338.025705] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3340.069853] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3358.177479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3363.128699] Lustre: server umount lustre-OST0000 complete [ 3370.657160] Lustre: server umount lustre-MDT0000 complete [ 3372.214225] LustreError: 211528:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144451 with bad export cookie 8661309805344360783 [ 3372.221060] LustreError: 211528:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3372.223458] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3372.397180] Lustre: server umount lustre-MDT0001 complete [ 3474.853779] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3476.055170] Key type lgssc unregistered [ 3476.195429] LNet: 214836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3476.197581] LNetError: 214836:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3476.208453] LNet: Removed LNI 192.168.201.109@tcp [ 3476.539143] Key type .llcrypt unregistered [ 3476.540865] Key type ._llcrypt unregistered [ 3485.153829] Key type ._llcrypt registered [ 3485.154907] Key type .llcrypt registered [ 3485.194990] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 3491.499849] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3491.849636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3491.888491] alg: No test for adler32 (adler32-zlib) [ 3492.752514] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 3492.852373] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3494.440185] Key type lgssc registered [ 3494.874855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3498.295186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3501.128141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3503.424401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3505.611122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3507.998087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3508.017165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3508.022952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3509.117397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3509.129523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3509.168641] Lustre: lustre-MDT0000: new disk, initializing [ 3509.196492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3509.205699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3510.580378] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3514.833856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3514.861381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3514.893774] Lustre: 218137:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3514.921587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3514.923246] Lustre: Skipped 1 previous similar message [ 3514.958599] Lustre: lustre-MDT0001: new disk, initializing [ 3514.995190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3515.009109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3515.012549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3516.391742] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3519.619171] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3520.790916] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3523.161134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3523.188572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3523.302952] Lustre: lustre-OST0000: new disk, initializing [ 3523.305221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3523.338126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.181960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3525.185742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3525.234328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3525.648511] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3528.890615] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3530.292361] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3530.364044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3531.697619] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3531.790589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3535.328946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3535.330951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.334695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3535.843772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3535.844429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.850269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3538.739253] LustreError: 221077:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3538.854656] Lustre: server umount lustre-OST0000 complete [ 3540.962333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3540.962687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3540.963158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3540.965962] Lustre: Skipped 1 previous similar message [ 3546.082033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3546.085370] Lustre: Skipped 2 previous similar messages [ 3546.560326] LustreError: 221279:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3546.563507] LustreError: 221279:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3546.599108] Lustre: server umount lustre-MDT0000 complete [ 3547.879869] LustreError: 218916:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144627 with bad export cookie 9580058601790348290 [ 3547.884383] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3547.887155] LustreError: 218916:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3547.968505] LustreError: 221481:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3547.970789] LustreError: 221481:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3548.069844] Lustre: server umount lustre-MDT0001 complete [ 3553.953675] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3557.548859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3557.712283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3559.033825] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3561.948303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3562.091119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3563.465336] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3564.445464] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3566.854728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3568.950445] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3571.948177] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3571.975053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3572.031489] Lustre: lustre-OST0001: new disk, initializing [ 3572.033227] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3572.062923] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3572.064768] Lustre: Skipped 1 previous similar message [ 3574.015243] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3576.844570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3576.848515] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3576.889447] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3579.546544] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3583.193353] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3591.958305] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:51:11 (1761144671) [ 3592.672726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3592.676145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3592.679757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3597.284051] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3597.287445] Lustre: Skipped 2 previous similar messages [ 3598.780841] LustreError: 226032:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3598.782663] LustreError: 226032:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3598.810518] Lustre: server umount lustre-MDT0000 complete [ 3600.006091] LustreError: 222632:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144679 with bad export cookie 9580058601790349158 [ 3600.009127] LustreError: 222632:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3600.009701] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3600.152131] Lustre: server umount lustre-MDT0001 complete [ 3611.636238] LustreError: 226436:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3611.638357] LustreError: 226436:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3611.655589] Lustre: server umount lustre-OST0000 complete [ 3623.155580] LustreError: 226637:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3623.158889] LustreError: 226637:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3623.249710] Lustre: server umount lustre-OST0001 complete [ 3625.411553] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 3628.797703] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3632.476865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3634.813786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3636.887741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3639.055500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3641.424795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3641.446794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3641.533607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3641.545758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3641.585897] Lustre: lustre-MDT0000: new disk, initializing [ 3641.614963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3641.622674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3642.899671] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3647.174494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3647.199035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3647.233847] Lustre: 228822:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3647.338258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3648.634067] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3651.715095] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3652.974749] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3655.371254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3655.401859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3655.506796] Lustre: lustre-OST0000: new disk, initializing [ 3655.508710] Lustre: Skipped 1 previous similar message [ 3655.510633] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3655.513118] Lustre: Skipped 2 previous similar messages [ 3657.540707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3657.542773] Lustre: Skipped 1 previous similar message [ 3657.545193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3657.585406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3657.743161] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3661.033212] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3662.615356] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3662.711833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3664.356591] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3664.440511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3667.940144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3667.940686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3667.941845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3667.941851] Lustre: Skipped 4 previous similar messages [ 3667.942469] LustreError: Skipped 1 previous similar message [ 3667.954361] Lustre: Skipped 1 previous similar message [ 3671.151099] LustreError: 231758:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3671.152910] LustreError: 231758:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3671.230471] Lustre: server umount lustre-OST0000 complete [ 3678.758561] Lustre: server umount lustre-MDT0000 complete [ 3680.014213] LustreError: 229601:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144759 with bad export cookie 9580058601790351258 [ 3680.017163] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3680.017567] LustreError: 229601:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3680.165089] Lustre: server umount lustre-MDT0001 complete [ 3683.049832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3683.217860] LustreError: 232638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3683.226575] LustreError: 232638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3683.244308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3683.245941] Lustre: Skipped 2 previous similar messages [ 3684.565595] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3687.437860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3688.906949] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3690.752300] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3691.964919] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3694.297776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3696.323684] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3698.066320] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3840.993023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3840.995075] LustreError: Skipped 1 previous similar message [ 3840.996630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3841.000848] Lustre: Skipped 2 previous similar messages [ 3841.002763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3841.004492] Lustre: Skipped 5 previous similar messages [ 3855.328196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3855.353716] LustreError: 234930:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3855.357515] LustreError: 234930:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3855.434133] Lustre: server umount lustre-OST0000 complete [ 3863.262092] Lustre: server umount lustre-MDT0000 complete [ 3864.799455] LustreError: 232619:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144944 with bad export cookie 9580058601790352140 [ 3864.803414] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3864.806030] LustreError: 232619:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3865.004791] Lustre: server umount lustre-MDT0001 complete [ 3962.600243] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3963.681924] Key type lgssc unregistered [ 3963.806405] LNet: 235899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3963.808817] LNetError: 235899:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3963.816301] LNet: Removed LNI 192.168.201.109@tcp [ 3964.126098] Key type .llcrypt unregistered [ 3964.127201] Key type ._llcrypt unregistered [ 3972.409713] Key type ._llcrypt registered [ 3972.410818] Key type .llcrypt registered [ 3972.445960] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 3979.000813] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3979.369694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3979.420589] alg: No test for adler32 (adler32-zlib) [ 3980.275830] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 3980.363188] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3981.952134] Key type lgssc registered [ 3982.297908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3985.413094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3987.674550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3989.792640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3992.069843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3997.704201] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4001.261411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4001.279828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4001.285352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4002.369830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4002.384174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4002.421329] Lustre: lustre-MDT0000: new disk, initializing [ 4002.447769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.457203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4003.780106] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4007.968250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4007.991206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4008.021144] Lustre: 239899:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4008.042864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4008.044500] Lustre: Skipped 1 previous similar message [ 4008.079105] Lustre: lustre-MDT0001: new disk, initializing [ 4008.112337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4008.124438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4008.127303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4009.501148] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4011.841500] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4014.027732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4014.052761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4014.160421] Lustre: lustre-OST0000: new disk, initializing [ 4014.162524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4014.198958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.046283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4015.050644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4015.111818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4016.260919] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4021.010427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4021.046310] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4021.117854] Lustre: lustre-OST0001: new disk, initializing [ 4021.120991] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4021.170935] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4023.224289] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4026.894862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4026.898322] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4026.945141] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4029.642344] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4032.921417] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4042.212814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.214857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4042.217895] Lustre: Skipped 3 previous similar messages [ 4042.221395] Lustre: Skipped 3 previous similar messages [ 4047.315254] LustreError: 243424:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4047.359750] Lustre: server umount lustre-MDT0000 complete [ 4048.594164] LustreError: 239886:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145128 with bad export cookie 16089384230188551176 [ 4048.597301] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4048.600590] LustreError: 239886:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4048.662522] LustreError: 243626:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4048.665488] LustreError: 243626:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4048.748878] Lustre: server umount lustre-MDT0001 complete [ 4059.649071] LustreError: 243826:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4059.654789] LustreError: 243826:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4059.690107] Lustre: server umount lustre-OST0000 complete [ 4071.156042] LustreError: 244028:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4071.158247] LustreError: 244028:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4071.231612] Lustre: server umount lustre-OST0001 complete [ 4075.889361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4078.637597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4078.972039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4081.684609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4084.701166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4085.047570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4091.112968] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4094.643787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.649441] Lustre: lustre-MDT0000: reset Object Index mappings [ 4094.808324] LustreError: 247334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4094.813256] LustreError: 247334:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4094.828948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4096.079336] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4098.977094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4098.984279] Lustre: lustre-MDT0001: reset Object Index mappings [ 4100.510260] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4101.439322] Lustre: 248444:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4103.712583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4103.839681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.843086] Lustre: Skipped 1 previous similar message [ 4104.869496] LustreError: 248984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4104.872364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4105.740991] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4108.603764] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4109.741431] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4110.725343] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4113.900682] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4115.072884] Lustre: 250291:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4124.641102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4124.643237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4124.647432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4127.232803] LustreError: 250396:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4127.235834] LustreError: 250396:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4127.275868] Lustre: server umount lustre-MDT0000 complete [ 4128.515257] LustreError: 247314:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145208 with bad export cookie 16089384230188557938 [ 4128.518380] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4128.518487] LustreError: 247314:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4128.679946] Lustre: server umount lustre-MDT0001 complete [ 4139.444800] LustreError: 250798:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4139.446908] LustreError: 250798:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4139.462313] Lustre: server umount lustre-OST0000 complete [ 4150.964023] Lustre: server umount lustre-OST0001 complete [ 4157.249019] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:00:37 (1761145237) [ 4162.361132] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4165.387585] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4168.710825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4171.005253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4173.009307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4175.180886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4177.544784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4177.573646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4177.665889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4177.682582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4177.724377] Lustre: lustre-MDT0000: new disk, initializing [ 4177.751768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4177.753820] Lustre: Skipped 1 previous similar message [ 4177.759817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4179.119722] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4183.847147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4183.884099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4183.939848] Lustre: 254516:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4184.087195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4185.498865] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4188.491840] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4189.625217] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4191.885750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4191.907737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4192.012662] Lustre: lustre-OST0000: new disk, initializing [ 4192.014817] Lustre: Skipped 1 previous similar message [ 4192.017428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4192.019923] Lustre: Skipped 2 previous similar messages [ 4193.350887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4193.353312] Lustre: Skipped 1 previous similar message [ 4193.355761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4193.409630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4194.014181] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4197.149294] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4198.687656] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4198.781170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4200.329946] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4200.401525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4203.489189] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4203.492117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4203.496597] Lustre: Skipped 3 previous similar messages [ 4203.498776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4203.500341] Lustre: Skipped 4 previous similar messages [ 4209.126260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4209.128231] Lustre: Skipped 2 previous similar messages [ 4214.243080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4214.244770] Lustre: Skipped 1 previous similar message [ 4215.264188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4215.288303] LustreError: 257468:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4215.292850] LustreError: 257468:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4215.383079] Lustre: server umount lustre-OST0000 complete [ 4219.361648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4219.364849] Lustre: Skipped 2 previous similar messages [ 4224.481227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4224.482889] Lustre: Skipped 3 previous similar messages [ 4231.136135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4231.206050] Lustre: server umount lustre-MDT0000 complete [ 4232.477474] LustreError: 255309:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145312 with bad export cookie 16089384230188560031 [ 4232.480436] LustreError: 255309:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4232.481494] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4232.652786] Lustre: server umount lustre-MDT0001 complete [ 4238.958567] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4242.265920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4242.403416] LustreError: 259047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4242.419094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.420961] Lustre: Skipped 2 previous similar messages [ 4243.687646] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4246.395625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4247.807959] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4248.785646] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4250.959484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4252.819780] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4255.498546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4255.522188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4255.573204] Lustre: lustre-OST0001: new disk, initializing [ 4255.575360] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4257.031957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4257.036408] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4257.068536] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4257.477423] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4261.794419] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4262.956314] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4269.606400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4269.611596] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4272.097743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4272.099787] LustreError: Skipped 1 previous similar message [ 4272.100900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4272.104360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4272.105858] Lustre: Skipped 2 previous similar messages [ 4277.380437] LustreError: 262552:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4277.382980] LustreError: 262552:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4277.415400] Lustre: server umount lustre-MDT0000 complete [ 4277.730782] LustreError: 259047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4277.736185] LustreError: 259047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4278.702598] LustreError: 259766:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145358 with bad export cookie 16089384230188560913 [ 4278.707983] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4278.708184] LustreError: 259766:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4278.839656] Lustre: server umount lustre-MDT0001 complete [ 4290.051268] Lustre: server umount lustre-OST0000 complete [ 4301.795483] Lustre: server umount lustre-OST0001 complete [ 4307.946592] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4311.687446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.823718] LustreError: 264332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4311.841278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4311.843100] Lustre: Skipped 3 previous similar messages [ 4313.332453] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4317.103590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4318.835836] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4320.005599] Lustre: 265440:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4323.068631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4325.459936] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4327.339434] LustreError: 265798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4327.356512] LustreError: 265798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4329.537489] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4331.349474] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4337.076679] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4337.085607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4340.113182] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4341.287307] Lustre: 267293:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4348.173334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4348.177659] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4351.379649] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:03:51 (1761145431) [ 4363.233890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4363.235882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.238770] Lustre: Skipped 3 previous similar messages [ 4363.240583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4363.242116] Lustre: Skipped 3 previous similar messages [ 4369.025981] LustreError: 268000:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4369.028223] LustreError: 268000:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4369.061690] Lustre: server umount lustre-MDT0000 complete [ 4370.236330] LustreError: 264313:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145450 with bad export cookie 16089384230188567199 [ 4370.242251] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4372.961050] LustreError: 267053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4372.965444] LustreError: 267053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4376.467329] Lustre: server umount lustre-MDT0001 complete [ 4383.737395] Lustre: server umount lustre-OST0000 complete [ 4391.018735] Lustre: server umount lustre-OST0001 complete [ 4393.160329] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4396.575950] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4400.377435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4402.970445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4404.888247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4407.077512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4413.313065] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4416.669348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4416.692527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4416.773520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4416.785212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4416.819953] Lustre: lustre-MDT0000: new disk, initializing [ 4416.851291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4418.148183] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4422.314395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4422.335595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4422.366573] Lustre: 271482:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4422.461623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4423.724431] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4428.136059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4428.156529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4428.497118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4430.003973] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4434.172038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4434.196615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4435.948718] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4436.134089] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4440.476288] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4450.757580] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 11:05:30 (1761145530) [ 4453.344854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4453.346991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4453.350089] Lustre: Skipped 5 previous similar messages [ 4453.351767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4453.353468] Lustre: Skipped 8 previous similar messages [ 4457.509646] Lustre: server umount lustre-MDT0000 complete [ 4458.748407] LustreError: 271473:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145538 with bad export cookie 16089384230188575053 [ 4458.751406] LustreError: 271473:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4458.751933] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4458.889487] Lustre: server umount lustre-MDT0001 complete [ 4469.763153] Lustre: server umount lustre-OST0000 complete [ 4481.289371] Lustre: server umount lustre-OST0001 complete [ 4487.397037] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4490.761645] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4494.017919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4496.256319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4498.222327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4500.316186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4502.911947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4502.935581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.024437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4503.027587] Lustre: Skipped 2 previous similar messages [ 4503.038165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4503.039775] Lustre: Skipped 4 previous similar messages [ 4503.076792] Lustre: lustre-MDT0000: new disk, initializing [ 4503.078030] Lustre: Skipped 3 previous similar messages [ 4503.101761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4503.103509] Lustre: Skipped 7 previous similar messages [ 4503.108993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4503.111090] Lustre: Skipped 3 previous similar messages [ 4504.476851] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4508.568070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4508.594124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4508.625065] Lustre: 278538:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4508.741122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4508.743773] Lustre: Skipped 2 previous similar messages [ 4510.116420] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4513.326310] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4514.564857] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4516.981470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4517.002883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4518.378838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4518.982200] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4522.171433] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4522.950147] Lustre: Mounted lustre-client [ 4523.270563] LustreError: 281039:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9a2b6e7f7800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4523.276405] LustreError: 281039:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4523.278662] LustreError: 281039:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 4523.302159] Lustre: Unmounted lustre-client [ 4528.099515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4528.102316] Lustre: Skipped 4 previous similar messages [ 4532.500714] Lustre: server umount lustre-OST0000 complete [ 4534.240965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4540.119451] Lustre: server umount lustre-MDT0000 complete [ 4541.420787] LustreError: 278531:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145621 with bad export cookie 16089384230188576579 [ 4541.425732] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4541.594665] Lustre: server umount lustre-MDT0001 complete [ 4545.775594] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4546.786552] Key type lgssc unregistered [ 4585.824670] LNet: 282317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4585.829529] LNetError: 282317:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4654.438669] LNet: Removed LNI 192.168.201.109@tcp [ 4654.842136] Key type .llcrypt unregistered [ 4654.843196] Key type ._llcrypt unregistered [ 4662.644362] Key type ._llcrypt registered [ 4662.645443] Key type .llcrypt registered [ 4662.687973] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4670.068558] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4670.450803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4670.467769] alg: No test for adler32 (adler32-zlib) [ 4671.317993] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 4671.398467] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4672.984205] Key type lgssc registered [ 4673.327749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4676.148196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4678.217121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4680.320723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4682.679260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4683.372269] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:09:23 (1761145763) [ 4685.736334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4685.758576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4685.763043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4686.872396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4686.886733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4686.924826] Lustre: lustre-MDT0000: new disk, initializing [ 4686.952979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4686.959656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4688.705166] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4693.817726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4693.844504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4693.879377] Lustre: 285487:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4693.913807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4693.915657] Lustre: Skipped 1 previous similar message [ 4693.951240] Lustre: lustre-MDT0001: new disk, initializing [ 4694.001397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4694.021194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4694.027085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4695.400566] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4698.564860] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4699.756213] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4702.466865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4702.508947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4702.657952] Lustre: lustre-OST0000: new disk, initializing [ 4702.659769] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4702.700384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4704.853701] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4708.293705] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4711.422491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4711.427256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4711.459641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4721.638175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4721.639195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4721.642845] LustreError: Skipped 1 previous similar message [ 4721.643777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4721.650920] Lustre: Skipped 1 previous similar message [ 4721.659331] Lustre: Skipped 1 previous similar message [ 4723.445165] LustreError: 287929:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4723.549347] Lustre: server umount lustre-OST0000 complete [ 4726.756418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4726.757618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4726.766775] Lustre: Skipped 1 previous similar message [ 4726.773750] Lustre: Skipped 1 previous similar message [ 4731.873115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4731.875067] Lustre: Skipped 1 previous similar message [ 4736.993537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4736.995672] Lustre: Skipped 1 previous similar message [ 4739.040132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4739.073391] LustreError: 288130:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4739.075456] LustreError: 288130:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4739.109937] Lustre: server umount lustre-MDT0000 complete [ 4740.432721] LustreError: 286265:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145820 with bad export cookie 14454205941719461008 [ 4740.437958] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4740.440713] LustreError: 286265:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4740.513966] LustreError: 288332:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4740.515927] LustreError: 288332:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4740.628513] Lustre: server umount lustre-MDT0001 complete [ 4744.498640] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4745.857442] Key type lgssc unregistered [ 4745.992435] LNet: 288864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4745.994534] LNetError: 288864:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4746.001434] LNet: Removed LNI 192.168.201.109@tcp [ 4746.342805] Key type .llcrypt unregistered [ 4746.344248] Key type ._llcrypt unregistered [ 4781.582543] Key type ._llcrypt registered [ 4781.583604] Key type .llcrypt registered [ 4781.626236] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4781.740463] Key type .llcrypt unregistered [ 4781.741959] Key type ._llcrypt unregistered [ 4809.171719] Key type ._llcrypt registered [ 4809.172753] Key type .llcrypt registered [ 4809.214957] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4809.346487] Key type .llcrypt unregistered [ 4809.347694] Key type ._llcrypt unregistered [ 4820.764357] Key type ._llcrypt registered [ 4820.766157] Key type .llcrypt registered [ 4820.804300] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4820.924714] Key type .llcrypt unregistered [ 4820.925898] Key type ._llcrypt unregistered [ 4849.898933] Key type ._llcrypt registered [ 4849.900281] Key type .llcrypt registered [ 4849.948645] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4850.065406] Key type .llcrypt unregistered [ 4850.066839] Key type ._llcrypt unregistered [ 4861.102515] Key type ._llcrypt registered [ 4861.103671] Key type .llcrypt registered [ 4861.152562] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4861.272413] Key type .llcrypt unregistered [ 4861.273679] Key type ._llcrypt unregistered [ 4872.923799] Key type ._llcrypt registered [ 4872.924727] Key type .llcrypt registered [ 4872.968341] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4873.086300] Key type .llcrypt unregistered [ 4873.087421] Key type ._llcrypt unregistered [ 4901.461688] Key type ._llcrypt registered [ 4901.462790] Key type .llcrypt registered [ 4901.505471] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4901.625385] Key type .llcrypt unregistered [ 4901.626370] Key type ._llcrypt unregistered [ 4912.937489] Key type ._llcrypt registered [ 4912.938462] Key type .llcrypt registered [ 4912.978645] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4913.096883] Key type .llcrypt unregistered [ 4913.098868] Key type ._llcrypt unregistered [ 4924.797052] Key type ._llcrypt registered [ 4924.798113] Key type .llcrypt registered [ 4924.843251] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4924.959477] Key type .llcrypt unregistered [ 4924.960591] Key type ._llcrypt unregistered [ 4944.301719] Key type ._llcrypt registered [ 4944.302956] Key type .llcrypt registered [ 4944.354440] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4950.780394] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4951.296747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4951.306448] alg: No test for adler32 (adler32-zlib) [ 4952.154528] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 4952.247078] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4953.832254] Key type lgssc registered [ 4954.246135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4957.986747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4960.812729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4963.134607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4965.400504] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4965.984270] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4824 sec ========== 11:14:05 (1761146045) [ 4966.536545] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:14:06 (1761146046) === [ 4967.681160] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:14:07 (1761146047) === [ 4981.186510] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4982.269667] Key type lgssc unregistered [ 4982.406447] LNet: 307651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4982.408545] LNetError: 307651:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4982.419421] LNet: Removed LNI 192.168.201.109@tcp [ 4982.745603] Key type .llcrypt unregistered [ 4982.746689] Key type ._llcrypt unregistered