[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 582089668 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003041] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008642] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010018] pid_max: default: 32768 minimum: 301 [ 0.011187] LSM: Security Framework initializing [ 0.012070] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014062] *** VALIDATE selinux *** [ 0.023438] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028700] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031128] *** VALIDATE tmpfs *** [ 0.033315] *** VALIDATE proc *** [ 0.034233] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.036280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037187] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040031] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.044184] debug: unmapping init [mem 0xffffffffa9459000-0xffffffffa9460fff] [ 0.047000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047715] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048024] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050009] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053016] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055308] rcu: Hierarchical SRCU implementation. [ 0.057698] smp: Bringing up secondary CPUs ... [ 0.058677] x86: Booting SMP configuration: [ 0.059025] .... node #0, CPUs: #1 #2 #3 [ 0.062168] smp: Brought up 1 node, 4 CPUs [ 0.063964] smpboot: Max logical packages: 1 [ 0.064010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134332] node 0 deferred pages initialised in 68ms [ 0.137327] devtmpfs: initialized [ 0.138193] x86/mm: Memory block size: 128MB [ 0.140333] gcov: version magic: 0x41383552 [ 0.142097] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144295] pinctrl core: initialized pinctrl subsystem [ 0.145139] [ 0.145436] ************************************************************* [ 0.146009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.147007] ** ** [ 0.148007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.149008] ** ** [ 0.150014] ** This means that this kernel is built to expose internal ** [ 0.151010] ** IOMMU data structures, which may compromise security on ** [ 0.152009] ** your system. ** [ 0.153009] ** ** [ 0.154012] ** If you see this message and you are not debugging the ** [ 0.155011] ** kernel, report this immediately to your vendor! ** [ 0.156009] ** ** [ 0.157007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158007] ************************************************************* [ 0.159733] NET: Registered protocol family 16 [ 0.160465] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.161069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.162065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.163418] cpuidle: using governor menu [ 0.164753] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.167490] PCI: Using configuration type 1 for base access [ 0.168124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.176092] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.178051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.181240] cryptd: max_cpu_qlen set to 1000 [ 0.184243] ACPI: Added _OSI(Module Device) [ 0.185016] ACPI: Added _OSI(Processor Device) [ 0.187013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.189013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.193423] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.200111] ACPI: Interpreter enabled [ 0.201118] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203015] ACPI: Using IOAPIC for interrupt routing [ 0.205157] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.208430] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.220054] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.222046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.226026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.231107] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236516] acpiphp: Slot [2] registered [ 0.238248] acpiphp: Slot [5] registered [ 0.239130] acpiphp: Slot [6] registered [ 0.241241] acpiphp: Slot [7] registered [ 0.242105] acpiphp: Slot [8] registered [ 0.243142] acpiphp: Slot [9] registered [ 0.245102] acpiphp: Slot [10] registered [ 0.246099] acpiphp: Slot [3] registered [ 0.248109] acpiphp: Slot [4] registered [ 0.249156] acpiphp: Slot [11] registered [ 0.251105] acpiphp: Slot [12] registered [ 0.252191] acpiphp: Slot [13] registered [ 0.254096] acpiphp: Slot [14] registered [ 0.255092] acpiphp: Slot [15] registered [ 0.257166] acpiphp: Slot [16] registered [ 0.258124] acpiphp: Slot [17] registered [ 0.260169] acpiphp: Slot [18] registered [ 0.261094] acpiphp: Slot [19] registered [ 0.263141] acpiphp: Slot [20] registered [ 0.264113] acpiphp: Slot [21] registered [ 0.266110] acpiphp: Slot [22] registered [ 0.267153] acpiphp: Slot [23] registered [ 0.269100] acpiphp: Slot [24] registered [ 0.270094] acpiphp: Slot [25] registered [ 0.271077] acpiphp: Slot [26] registered [ 0.272076] acpiphp: Slot [27] registered [ 0.273197] acpiphp: Slot [28] registered [ 0.275070] acpiphp: Slot [29] registered [ 0.276088] acpiphp: Slot [30] registered [ 0.277097] acpiphp: Slot [31] registered [ 0.278041] PCI host bridge to bus 0000:00 [ 0.279017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.281014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.282014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.284017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.285015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.287017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.290993] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.294266] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.304017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.309066] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.313020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.315020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.318019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.321083] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.322568] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.324034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.326870] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.330012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.340019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.344019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.348157] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.353030] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.360018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.376024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.383784] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.392017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.397021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.413042] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.427154] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.436030] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.445019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.475024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.488534] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.506019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.516028] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.545019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.560324] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.567018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.572019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.583016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.597255] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.603019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.608016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.625017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.636913] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.639366] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.641384] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.643337] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.645240] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.651055] iommu: Default domain type: Passthrough [ 0.653474] SCSI subsystem initialized [ 0.655145] ACPI: bus type USB registered [ 0.656120] usbcore: registered new interface driver usbfs [ 0.658076] usbcore: registered new interface driver hub [ 0.660115] usbcore: registered new device driver usb [ 0.662175] pps_core: LinuxPPS API ver. 1 registered [ 0.664023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.667064] PTP clock support registered [ 0.669085] EDAC MC: Ver: 3.0.0 [ 0.671114] PCI: Using ACPI for IRQ routing [ 0.672947] NetLabel: Initializing [ 0.675013] NetLabel: domain hash size = 128 [ 0.676011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.678088] NetLabel: unlabeled traffic allowed by default [ 0.680139] vgaarb: loaded [ 0.682290] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.685020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.692052] clocksource: Switched to clocksource kvm-clock [ 0.795718] VFS: Disk quotas dquot_6.6.0 [ 0.796785] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.798259] *** VALIDATE ramfs *** [ 0.799201] *** VALIDATE hugetlbfs *** [ 0.800307] pnp: PnP ACPI init [ 0.801835] pnp: PnP ACPI: found 6 devices [ 0.815819] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.819367] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.821781] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.823742] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.825315] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.826904] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.829448] NET: Registered protocol family 2 [ 0.832128] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.837698] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.840191] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.844470] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.847456] TCP: Hash tables configured (established 65536 bind 65536) [ 0.849416] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.851303] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.853113] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.855030] NET: Registered protocol family 1 [ 0.857302] RPC: Registered named UNIX socket transport module. [ 0.859499] RPC: Registered udp transport module. [ 0.861177] RPC: Registered tcp transport module. [ 0.862733] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.865459] NET: Registered protocol family 44 [ 0.867390] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.869402] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.871053] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.872527] PCI: CLS 0 bytes, default 64 [ 0.874156] Unpacking initramfs... [ 2.217326] debug: unmapping init [mem 0xffff98bb3cc54000-0xffff98bb3ffbffff] [ 2.221502] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.223857] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.226816] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.746637] Initialise system trusted keyrings [ 2.748377] Key type blacklist registered [ 2.750286] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.761153] zbud: loaded [ 2.764518] *** VALIDATE nfs *** [ 2.765993] *** VALIDATE nfs4 *** [ 2.767857] pstore: using deflate compression [ 2.772400] Platform Keyring initialized [ 2.879969] NET: Registered protocol family 38 [ 2.882487] Key type asymmetric registered [ 2.884628] Asymmetric key parser 'x509' registered [ 2.886972] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.890316] io scheduler mq-deadline registered [ 2.892371] io scheduler kyber registered [ 2.894510] io scheduler bfq registered [ 2.897135] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.899689] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.902115] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.904503] ACPI: Power Button [PWRF] [ 2.909423] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.915163] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.925291] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.931186] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.947829] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.973853] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.001502] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.006909] Non-volatile memory driver v1.3 [ 3.008959] Linux agpgart interface v0.103 [ 3.045082] virtio_blk virtio1: [vda] 146136 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.048214] vda: detected capacity change from 0 to 74821632 [ 3.071839] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.074613] vdb: detected capacity change from 0 to 1073741824 [ 3.093353] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.097199] vdc: detected capacity change from 0 to 2621440000 [ 3.120857] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.124119] vdd: detected capacity change from 0 to 2621440000 [ 3.143956] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.146994] vde: detected capacity change from 0 to 4294967296 [ 3.163383] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.166843] vdf: detected capacity change from 0 to 4294967296 [ 3.195041] libphy: Fixed MDIO Bus: probed [ 3.202848] usbcore: registered new interface driver usbserial_generic [ 3.205750] usbserial: USB Serial support registered for generic [ 3.208826] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.214222] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.216226] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.219197] mousedev: PS/2 mouse device common for all mice [ 3.223759] rtc_cmos 00:05: RTC can wake from S4 [ 3.227049] rtc_cmos 00:05: registered as rtc0 [ 3.229833] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.230509] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.233675] intel_pstate: CPU model not supported [ 3.239208] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.243402] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.243404] hid: raw HID events driver (C) Jiri Kosina [ 3.248812] usbcore: registered new interface driver usbhid [ 3.251031] usbhid: USB HID core driver [ 3.252763] drop_monitor: Initializing network drop monitor service [ 3.255515] Initializing XFRM netlink socket [ 3.257749] NET: Registered protocol family 10 [ 3.260728] Segment Routing with IPv6 [ 3.262368] NET: Registered protocol family 17 [ 3.264638] mpls_gso: MPLS GSO support [ 3.270693] RAS: Correctable Errors collector initialized. [ 3.272678] AVX version of gcm_enc/dec engaged. [ 3.274238] AES CTR mode by8 optimization enabled [ 3.352611] sched_clock: Marking stable (3352562014, 0)->(4278955489, -926393475) [ 3.357174] registered taskstats version 1 [ 3.359225] Loading compiled-in X.509 certificates [ 3.360790] zswap: loaded using pool lzo/zbud [ 3.382725] Key type big_key registered [ 3.395600] Key type encrypted registered [ 3.396938] ima: No TPM chip found, activating TPM-bypass! [ 3.398629] ima: Allocated hash algorithm: sha1 [ 3.399845] ima: No architecture policies found [ 3.401244] evm: Initialising EVM extended attributes: [ 3.402716] evm: security.selinux [ 3.403702] evm: security.ima [ 3.404439] evm: security.capability [ 3.405435] evm: HMAC attrs: 0x1 [ 3.407401] rtc_cmos 00:05: setting system clock to 2026-08-23 04:36:40 UTC (1787459800) [ 3.412919] debug: unmapping init [mem 0xffffffffaa403000-0xffffffffaa5fffff] [ 3.415236] debug: unmapping init [mem 0xffffffffa9182000-0xffffffffa9458fff] [ 3.422098] Write protecting the kernel read-only data: 28672k [ 3.424325] debug: unmapping init [mem 0xffffffffa7803000-0xffffffffa79fffff] [ 3.426214] debug: unmapping init [mem 0xffffffffa8114000-0xffffffffa81fffff] [ 3.452962] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.458658] systemd[1]: Detected virtualization kvm. [ 3.459830] systemd[1]: Detected architecture x86-64. [ 3.461091] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.493753] systemd[1]: No hostname configured. [ 3.494966] systemd[1]: Set hostname to . [ 3.496613] random: systemd: uninitialized urandom read (16 bytes read) [ 3.498310] systemd[1]: Initializing machine ID from random generator. [ 3.623975] random: systemd: uninitialized urandom read (16 bytes read) [ 3.626602] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.630392] random: systemd: uninitialized urandom read (16 bytes read) [ 3.633225] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.642762] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.201235] device-mapper: uevent: version 1.0.3 [ 4.203155] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.874985] virtio_net virtio0 ens2: renamed from eth0 [ 4.878180] random: fast init done [ 4.917256] scsi host0: ata_piix [ 4.936021] scsi host1: ata_piix [ 4.938529] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.941501] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.929787] random: crng init done [ 9.933437] random: 7 urandom warning(s) missed due to ratelimiting [ 9.987778] dracut-initqueue[578]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.350864] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.617194] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.552425] SELinux: Disabled at runtime. [ 16.685574] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.716612] systemd[1]: Detected virtualization kvm. [ 16.719634] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.754504] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.774342] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.799843] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.808804] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.826698] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.855475] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.879436] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Pro[ 19.122352] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS cess Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.471647] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.566023] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.628125] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.353176] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.485383] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 27.957710] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 28.824694] NFS: Registering the id_resolver key type [ 28.828690] Key type id_resolver registered [ 28.839636] Key type id_legacy registered [ **] A start job is running for Configur…only root support (10s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg118-server login: [ 70.246171] hrtimer: interrupt took 7990204 ns [ 103.027248] libcfs: loading out-of-tree module taints kernel. [ 103.075949] Key type ._llcrypt registered [ 103.084901] Key type .llcrypt registered [ 103.253300] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 122.903549] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 125.160257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 125.180861] alg: No test for adler32 (adler32-zlib) [ 126.611944] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 127.644328] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 129.463214] Key type lgssc registered [ 131.719986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 151.102133] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 205.784115] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 220.834543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 220.886081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.260547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 222.336370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 222.496125] Lustre: lustre-MDT0000: new disk, initializing [ 222.599858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.643550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 227.561975] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.339834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 243.523199] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 243.569570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 243.581694] Lustre: Skipped 1 previous similar message [ 243.794520] Lustre: lustre-MDT0001: new disk, initializing [ 243.939329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.018698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 244.033566] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 248.880935] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.921445] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 267.715256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.994585] Lustre: lustre-OST0000: new disk, initializing [ 268.000714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 268.012215] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 268.088573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 275.643957] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 277.530110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 277.562319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 277.687899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 292.993366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 293.389922] Lustre: lustre-OST0001: new disk, initializing [ 293.395248] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 293.415456] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 293.548279] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 299.585911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 299.596282] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 299.717456] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 302.127283] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 318.301440] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 327.490733] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 334.903817] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing check_logdir /tmp/testlogs/ [ 340.865166] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing yml_node [ 345.835900] Lustre: DEBUG MARKER: Client: 2.17.57.65 [ 348.705599] Lustre: DEBUG MARKER: MDS: 2.17.57.65 [ 351.956969] Lustre: DEBUG MARKER: OSS: 2.17.57.65 [ 353.596406] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lfsck ============----- Sun Aug 23 00:42:30 EDT 2026 [ 372.603462] Lustre: DEBUG MARKER: excepting tests: 18b 23b [ 383.253831] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 412.640175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 412.647359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 414.664528] Lustre: server umount lustre-MDT0000 complete [ 422.882447] LustreError: 9521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 422.893247] LustreError: 9521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 424.538400] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787460221 with bad export cookie 11792166018848995902 [ 424.544144] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 424.547787] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 424.836968] Lustre: server umount lustre-MDT0001 complete [ 444.390245] Lustre: 3644:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787460225/real 1787460225] req@ffff98ba87d82d80 x1874287577385216/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787460241 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 444.414611] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.426077] Lustre: Skipped 3 previous similar messages [ 444.442049] Lustre: server umount lustre-OST0000 complete [ 446.432972] Lustre: 3641:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787460227/real 1787460227] req@ffff98ba86bfd180 x1874287577385472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787460243 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 448.545082] Lustre: 3644:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787460229/real 1787460229] req@ffff98ba86bfe300 x1874287577385728/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787460245 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 451.551120] Lustre: 3642:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787460232/real 1787460232] req@ffff98bbb79fc700 x1874287577386112/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787460248 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 453.748212] Lustre: server umount lustre-OST0001 complete [ 473.997407] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 478.070912] Key type lgssc unregistered [ 478.541162] LNet: 14823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 478.553293] LNetError: 14823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 478.587268] LNet: Removed LNI 192.168.201.118@tcp [ 479.717138] Key type .llcrypt unregistered [ 479.723461] Key type ._llcrypt unregistered [ 509.339348] Key type ._llcrypt registered [ 509.340992] Key type .llcrypt registered [ 509.464580] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 527.313722] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 528.865394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 529.017288] alg: No test for adler32 (adler32-zlib) [ 530.202528] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 530.575451] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 532.295198] Key type lgssc registered [ 534.008889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 591.572586] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 605.778068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 605.831370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.322340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 607.392860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 607.590767] Lustre: lustre-MDT0000: new disk, initializing [ 607.727948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 607.772828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 613.615413] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 629.785223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.958273] Lustre: 19281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 630.009389] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 630.016718] Lustre: Skipped 1 previous similar message [ 630.156795] Lustre: lustre-MDT0001: new disk, initializing [ 630.303824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 630.349496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 630.367717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 635.488784] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 641.654497] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 654.399259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 654.651918] Lustre: lustre-OST0000: new disk, initializing [ 654.654831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 654.667351] Lustre: 21219:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 654.747610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 655.242120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 655.265882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 655.466679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 662.976284] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 681.574742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 681.703782] Lustre: lustre-OST0001: new disk, initializing [ 681.710246] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 681.715249] Lustre: 22248:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 681.828667] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 687.669681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 687.685964] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 687.758536] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 689.829265] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 704.817410] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 713.477867] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 723.475279] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 00:48:39 (1787460519) === [ 727.935788] Lustre: DEBUG MARKER: == sanity-lfsck test 0: Control LFSCK manually =========== 00:48:43 (1787460523) [ 728.190826] Lustre: 21685:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 728.211054] Lustre: 21685:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 728.220357] Lustre: 21685:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 728.237239] Lustre: 21685:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 728.259933] Lustre: 21685:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 728.273967] Lustre: 21685:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 728.788100] Lustre: 19287:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 728.798910] Lustre: 19287:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 8 previous similar messages [ 728.810931] Lustre: 19287:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 728.815567] Lustre: 19287:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 728.820516] Lustre: 19287:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 728.831030] Lustre: 19287:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 728.837634] Lustre: 19287:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 728.848285] Lustre: 19287:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 728.861541] Lustre: 19287:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 728.872681] Lustre: 19287:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 728.884509] Lustre: 19287:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 728.894508] Lustre: 19287:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 8 previous similar messages [ 729.854808] Lustre: 21685:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 729.875178] Lustre: 21685:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 23 previous similar messages [ 729.888084] Lustre: 21685:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 729.901780] Lustre: 21685:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 729.919502] Lustre: 21685:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 729.940953] Lustre: 21685:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 729.963901] Lustre: 21685:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 729.977470] Lustre: 21685:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 729.990723] Lustre: 21685:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 730.000796] Lustre: 21685:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 730.017755] Lustre: 21685:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 730.021121] Lustre: 21685:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 23 previous similar messages [ 731.956989] Lustre: 21685:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 731.978926] Lustre: 21685:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 65 previous similar messages [ 731.982802] Lustre: 21685:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 731.996408] Lustre: 21685:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 65 previous similar messages [ 732.002614] Lustre: 21685:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 732.020904] Lustre: 21685:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 65 previous similar messages [ 732.029165] Lustre: 21685:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 1/3/0 [ 732.033223] Lustre: 21685:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 65 previous similar messages [ 732.040132] Lustre: 21685:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 732.047628] Lustre: 21685:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 65 previous similar messages [ 732.056803] Lustre: 21685:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 732.066438] Lustre: 21685:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 65 previous similar messages [ 736.055785] Lustre: 19289:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 257 < left 264, rollback = 2 [ 736.076860] Lustre: 19289:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 139 previous similar messages [ 736.089391] Lustre: 19289:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 736.100815] Lustre: 19289:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 139 previous similar messages [ 736.109113] Lustre: 19289:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 736.115596] Lustre: 19289:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 139 previous similar messages [ 736.125289] Lustre: 19289:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 736.140944] Lustre: 19289:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 139 previous similar messages [ 736.160405] Lustre: 19289:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 736.172345] Lustre: 19289:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 139 previous similar messages [ 736.180807] Lustre: 19289:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 736.193510] Lustre: 19289:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 139 previous similar messages [ 738.760699] Lustre: *** cfs_fail_loc=1600, val=3*** [ 741.791215] Lustre: *** cfs_fail_loc=1600, val=3*** [ 744.126898] Lustre: *** cfs_fail_loc=1600, val=3*** [ 754.294857] Lustre: 23546:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 274, rollback = 2 [ 754.305303] Lustre: 23546:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 80 previous similar messages [ 754.308877] Lustre: 21210:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 754.322369] Lustre: 23546:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/274/0 [ 754.330771] Lustre: 21210:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 81 previous similar messages [ 754.330807] Lustre: 21210:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 754.330811] Lustre: 21210:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 80 previous similar messages [ 754.330817] Lustre: 21210:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 754.330821] Lustre: 21210:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 80 previous similar messages [ 754.330826] Lustre: 21210:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 754.330828] Lustre: 21210:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 80 previous similar messages [ 754.400838] Lustre: 23546:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 82 previous similar messages [ 758.243380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 758.255815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 758.265575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 759.265631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 759.265778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 759.292021] Lustre: Skipped 1 previous similar message [ 759.292401] Lustre: Skipped 1 previous similar message [ 763.711324] Lustre: server umount lustre-MDT0000 complete [ 767.783873] LustreError: 19274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787460564 with bad export cookie 259145768625451996 [ 767.788565] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 767.804918] LustreError: 19274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 768.238882] Lustre: server umount lustre-MDT0001 complete [ 782.280514] Lustre: server umount lustre-OST0000 complete [ 785.375274] Lustre: 16439:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787460566/real 1787460566] req@ffff98ba87cd1500 x1874287999965824/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787460582 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 785.413416] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 785.431683] Lustre: Skipped 1 previous similar message [ 787.298912] Lustre: server umount lustre-OST0001 complete [ 797.723870] Lustre: DEBUG MARKER: == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================================================== 00:49:53 (1787460593) [ 814.356884] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 827.690948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 828.351901] LustreError: 26261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 828.374745] LustreError: 26261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 828.465276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 833.509630] LustreError: 26262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 834.805431] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 838.626426] LustreError: 26261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 843.747494] LustreError: 26262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 846.815404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.103446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 852.476440] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 855.684379] Lustre: 27402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 863.510837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 863.913150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 872.107375] LustreError: 28083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 872.896957] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 873.155216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 883.695176] LustreError: 27755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 883.728187] LustreError: 27755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 885.270992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 890.862093] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:40 to 0x2c0000401:65) [ 893.990658] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 902.945293] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 907.324838] Lustre: 29276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 909.419627] Lustre: 27985:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 909.444257] Lustre: 27985:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 25 previous similar messages [ 909.455153] Lustre: 27985:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 909.464474] Lustre: 27985:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 25 previous similar messages [ 909.476356] Lustre: 27985:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 909.487910] Lustre: 27985:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 24 previous similar messages [ 909.491217] Lustre: 27985:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 909.519249] Lustre: 27985:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 26 previous similar messages [ 909.533951] Lustre: 27985:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 909.545890] Lustre: 27985:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 26 previous similar messages [ 909.551460] Lustre: 27985:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 909.558887] Lustre: 27985:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 26 previous similar messages [ 915.886442] Lustre: *** cfs_fail_loc=1501, val=0*** [ 925.377280] Lustre: Failing over lustre-MDT0000 [ 925.893141] Lustre: server umount lustre-MDT0000 complete [ 926.691457] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.693312] LustreError: 26257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.711318] Lustre: Skipped 2 previous similar messages [ 926.738659] LustreError: 26257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 938.327928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 938.462511] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 938.883081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 938.890961] Lustre: Skipped 1 previous similar message [ 938.928951] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 944.097066] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 944.099204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 944.156380] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 944.243099] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:105 to 0x2c0000401:129) [ 944.247056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:129) [ 944.505883] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 948.941036] Lustre: *** cfs_fail_loc=1505, val=0*** [ 958.177686] Lustre: DEBUG MARKER: == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ========================================================== 00:52:34 (1787460754) [ 959.650708] Lustre: 27985:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 959.660221] Lustre: 27985:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 322 previous similar messages [ 959.666186] Lustre: 27985:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 959.672276] Lustre: 27985:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 959.678701] Lustre: 27985:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 959.684994] Lustre: 27985:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 959.692053] Lustre: 27985:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 959.697770] Lustre: 27985:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 321 previous similar messages [ 959.704540] Lustre: 27985:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 959.710494] Lustre: 27985:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 959.716469] Lustre: 27985:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 959.722185] Lustre: 27985:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 322 previous similar messages [ 966.582489] Lustre: *** cfs_fail_loc=1502, val=0*** [ 978.192200] Lustre: Failing over lustre-MDT0000 [ 978.414900] Lustre: server umount lustre-MDT0000 complete [ 979.941342] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.955565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 979.957800] Lustre: Skipped 3 previous similar messages [ 979.958376] LustreError: 29365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.958384] LustreError: 29365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 990.303491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 990.405445] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 990.719873] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 995.454807] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 995.807805] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 995.817171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 995.828797] Lustre: Skipped 3 previous similar messages [ 995.875582] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 995.928949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 995.932187] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:169 to 0x2c0000401:193) [ 999.284378] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1006.964933] Lustre: DEBUG MARKER: == sanity-lfsck test 1c: LFSCK can find out and repair lost FID-in-dirent ========================================================== 00:53:23 (1787460803) [ 1014.544266] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1014.551840] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1014.562449] Lustre: Skipped 1 previous similar message [ 1024.295923] Lustre: Failing over lustre-MDT0000 [ 1024.642850] Lustre: server umount lustre-MDT0000 complete [ 1026.530291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.549110] Lustre: Skipped 3 previous similar messages [ 1036.574298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1036.741481] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1037.036081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1037.042907] Lustre: Skipped 1 previous similar message [ 1037.081487] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1041.789269] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1042.407879] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1042.417254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1042.427916] Lustre: Skipped 3 previous similar messages [ 1042.454895] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1042.507939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:233 to 0x280000401:257) [ 1042.511277] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:233 to 0x2c0000401:257) [ 1045.291989] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1054.667337] Lustre: DEBUG MARKER: == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================================================== 00:54:10 (1787460850) [ 1056.126059] Lustre: 26258:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1056.139960] Lustre: 26258:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 644 previous similar messages [ 1056.150541] Lustre: 26258:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1056.162576] Lustre: 26258:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 1056.183388] Lustre: 26258:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1056.191981] Lustre: 26258:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 643 previous similar messages [ 1056.199925] Lustre: 26258:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1056.206124] Lustre: 26258:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 1056.212859] Lustre: 26258:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1056.218851] Lustre: 26258:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 1056.225830] Lustre: 26258:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1056.231775] Lustre: 26258:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 644 previous similar messages [ 1062.886921] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1073.854688] Lustre: Failing over lustre-MDT0000 [ 1074.147598] Lustre: server umount lustre-MDT0000 complete [ 1078.239880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1078.243695] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.245022] LustreError: 29365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1078.245031] LustreError: 29365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 1078.371842] Lustre: Skipped 4 previous similar messages [ 1090.047167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1090.239300] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1090.610411] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1095.649049] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1095.671085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1095.686363] Lustre: Skipped 3 previous similar messages [ 1095.711903] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1095.779306] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:297 to 0x2c0000401:321) [ 1095.784232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:297 to 0x280000401:321) [ 1096.617809] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1105.916658] Lustre: DEBUG MARKER: == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================================================== 00:55:02 (1787460902) [ 1112.011439] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1120.833924] Lustre: Failing over lustre-MDT0000 [ 1121.249397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.250695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1121.265565] Lustre: Skipped 2 previous similar messages [ 1121.296683] Lustre: server umount lustre-MDT0000 complete [ 1133.113929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1133.188612] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1133.460828] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1138.658317] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1138.672842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1138.683926] Lustre: Skipped 3 previous similar messages [ 1138.698762] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1138.714415] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1138.769272] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:361 to 0x2c0000401:385) [ 1138.770238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:361 to 0x280000401:385) [ 1148.623091] Lustre: DEBUG MARKER: == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================================================== 00:55:45 (1787460945) [ 1154.834487] Lustre: *** cfs_fail_loc=1605, val=0*** [ 1163.758960] Lustre: Failing over lustre-MDT0000 [ 1164.206992] Lustre: server umount lustre-MDT0000 complete [ 1175.941821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1176.084596] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1176.367640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.372460] Lustre: Skipped 2 previous similar messages [ 1176.427808] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1181.520824] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1181.665070] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1181.667806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1181.682741] Lustre: Skipped 3 previous similar messages [ 1181.721909] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1181.801037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:425 to 0x280000401:449) [ 1181.801922] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:425 to 0x2c0000401:449) [ 1190.668949] Lustre: DEBUG MARKER: == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ========================================================== 00:56:26 (1787460986) [ 1192.191042] Lustre: 26257:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1192.197671] Lustre: 26257:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 968 previous similar messages [ 1192.203467] Lustre: 26257:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1192.211318] Lustre: 26257:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1192.216251] Lustre: 26257:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1192.225732] Lustre: 26257:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1192.235088] Lustre: 26257:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1192.243624] Lustre: 26257:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1192.251685] Lustre: 26257:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 1192.266723] Lustre: 26257:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1192.275327] Lustre: 26257:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1192.286930] Lustre: 26257:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 968 previous similar messages [ 1197.678841] Lustre: *** cfs_fail_loc=161d, val=0*** [ 1207.099093] Lustre: Failing over lustre-MDT0000 [ 1207.264106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.265995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1207.292135] Lustre: Skipped 6 previous similar messages [ 1207.313254] Lustre: Skipped 2 previous similar messages [ 1207.527793] Lustre: server umount lustre-MDT0000 complete [ 1212.385553] LustreError: 26257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.438975] LustreError: 26257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 1220.532357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1220.682488] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1220.995236] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1226.218903] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1226.227811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1226.250757] Lustre: Skipped 3 previous similar messages [ 1226.280443] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1226.335571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:489 to 0x280000401:513) [ 1226.340176] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 1227.420732] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1240.910366] Lustre: DEBUG MARKER: == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ========================================================== 00:57:16 (1787461036) [ 1243.671297] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1255.775764] Lustre: DEBUG MARKER: == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ========================================================== 00:57:32 (1787461052) [ 1262.890574] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1264.054134] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1275.339562] Lustre: DEBUG MARKER: == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ========================================================== 00:57:51 (1787461071) [ 1310.959181] Lustre: Failing over lustre-MDT0000 [ 1311.305951] Lustre: server umount lustre-MDT0000 complete [ 1313.249845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1317.378851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1327.408265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1329.640399] Lustre: 16439:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787461110/real 1787461110] req@ffff98ba8528e680 x1874288000627072/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787461126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1329.682209] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1339.904266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1339.930878] Lustre: lustre-MDT0000: reset Object Index mappings [ 1354.212896] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x398abb41bf951b4 [ 1354.227945] Lustre: MGC192.168.201.118@tcp: Connection restored to 0@lo (at 0@lo) [ 1354.234993] Lustre: Skipped 3 previous similar messages [ 1354.628242] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1359.617857] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1359.849883] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1359.905888] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1359.948695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:609) [ 1359.950705] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:609) [ 1363.446336] LustreError: 42926:0:(lfsck_engine.c:1018:lfsck_master_engine()) lustre-MDT0000-osd: master engine fail to verify the .lustre/lost+found/, go ahead: rc = -115 [ 1363.471288] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1364.515060] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1365.536870] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1367.585484] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1367.587280] Lustre: Skipped 1 previous similar message [ 1373.428014] Lustre: Failing over lustre-MDT0000 [ 1373.799863] Lustre: server umount lustre-MDT0000 complete [ 1375.208348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.211395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1375.223088] Lustre: Skipped 7 previous similar messages [ 1384.586295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1389.491600] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1390.123977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:592 to 0x280000401:641) [ 1390.131361] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:592 to 0x2c0000401:641) [ 1393.604465] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1401.990878] Lustre: DEBUG MARKER: == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ========================================================== 00:59:57 (1787461197) [ 1404.533817] Lustre: *** cfs_fail_loc=1504, val=0*** [ 1414.560868] Lustre: Failing over lustre-MDT0000 [ 1414.851856] Lustre: server umount lustre-MDT0000 complete [ 1415.647688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1420.422266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1430.041463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1430.994579] Lustre: 16439:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787461212/real 1787461212] req@ffff98bbc12dfb80 x1874288000726272/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787461228 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1441.865663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1441.886875] Lustre: lustre-MDT0000: reset Object Index mappings [ 1456.830390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.834255] Lustre: Skipped 3 previous similar messages [ 1456.862165] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1456.870185] Lustre: Skipped 1 previous similar message [ 1462.244072] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1462.255748] Lustre: Skipped 1 previous similar message [ 1462.273539] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1462.280919] Lustre: Skipped 8 previous similar messages [ 1462.306873] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1462.315932] Lustre: Skipped 1 previous similar message [ 1462.385214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:705) [ 1462.387674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:705) [ 1462.485278] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1466.566052] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1474.783313] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1474.785190] Lustre: Skipped 7 previous similar messages [ 1480.530200] Lustre: Failing over lustre-MDT0000 [ 1480.766371] Lustre: server umount lustre-MDT0000 complete [ 1482.723243] LustreError: 26261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.743415] LustreError: 26261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 91 previous similar messages [ 1490.488752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1490.605181] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1490.625969] LustreError: Skipped 2 previous similar messages [ 1495.538854] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1496.101964] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:681 to 0x2c0000401:737) [ 1496.103184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:681 to 0x280000401:737) [ 1498.888038] Lustre: *** cfs_fail_loc=1505, val=0*** [ 1498.892889] Lustre: Skipped 84 previous similar messages [ 1505.735769] Lustre: DEBUG MARKER: == sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ========================================================== 01:01:42 (1787461302) [ 1506.925516] Lustre: 26258:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 278, rollback = 2 [ 1506.929876] Lustre: 26258:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1305 previous similar messages [ 1506.933588] Lustre: 26258:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 1506.937922] Lustre: 26258:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1305 previous similar messages [ 1506.941971] Lustre: 26258:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1506.948129] Lustre: 26258:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1305 previous similar messages [ 1506.959129] Lustre: 26258:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1506.974285] Lustre: 26258:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1305 previous similar messages [ 1506.988270] Lustre: 26258:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 1506.995204] Lustre: 26258:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1306 previous similar messages [ 1507.002838] Lustre: 26258:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1507.012870] Lustre: 26258:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1305 previous similar messages [ 1513.334164] Lustre: *** cfs_fail_loc=1600, val=1*** [ 1534.656973] Lustre: DEBUG MARKER: == sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ========================================================== 01:02:10 (1787461330) [ 1545.830396] Lustre: *** cfs_fail_loc=1601, val=1*** [ 1545.832129] Lustre: Skipped 10 previous similar messages [ 1571.760493] Lustre: DEBUG MARKER: == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ========================================================== 01:02:48 (1787461368) [ 1587.754056] Lustre: Failing over lustre-MDT0000 [ 1587.967202] Lustre: server umount lustre-MDT0000 complete [ 1588.191643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1596.460603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1596.838749] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1596.843649] Lustre: Skipped 1 previous similar message [ 1601.281562] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1602.025462] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1602.027315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1602.040976] Lustre: Skipped 1 previous similar message [ 1602.056980] Lustre: Skipped 7 previous similar messages [ 1602.071731] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1602.080206] Lustre: Skipped 1 previous similar message [ 1602.122279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:854 to 0x280000401:897) [ 1602.129780] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:853 to 0x2c0000401:897) [ 1610.573345] Lustre: DEBUG MARKER: == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ========================================================== 01:03:27 (1787461407) [ 1623.948784] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1642.470727] Lustre: 52953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1667.345708] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1671.070228] Lustre: 54091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1678.068247] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1678.070040] Lustre: Skipped 81 previous similar messages [ 1680.605449] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1681.631124] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1682.655212] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1684.708142] Lustre: *** cfs_fail_loc=1602, val=1*** [ 1684.712756] Lustre: Skipped 1 previous similar message [ 1685.422628] Lustre: Failing over lustre-MDT0000 [ 1685.681764] Lustre: server umount lustre-MDT0000 complete [ 1689.065436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1689.073038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1689.093734] Lustre: Skipped 15 previous similar messages [ 1694.757617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1700.122566] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1700.457666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:947 to 0x280000401:993) [ 1700.458578] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:946 to 0x2c0000401:961) [ 1709.743775] Lustre: DEBUG MARKER: == sanity-lfsck test 8: LFSCK state machine ============== 01:05:06 (1787461506) [ 1715.684702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1715.698169] Lustre: Skipped 6 previous similar messages [ 1718.221348] Lustre: server umount lustre-MDT0000 complete [ 1721.889185] LustreError: 26243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787461518 with bad export cookie 259145768625666987 [ 1721.917234] LustreError: 26243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1722.303150] Lustre: server umount lustre-MDT0001 complete [ 1736.684609] Lustre: server umount lustre-OST0000 complete [ 1750.901917] Lustre: server umount lustre-OST0001 complete [ 1758.322626] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 1766.773920] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1813.284401] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1825.266430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1825.608322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1825.675653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1825.813938] Lustre: lustre-MDT0000: new disk, initializing [ 1825.968994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1830.688441] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1842.118359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.272531] Lustre: 59151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1842.352488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1842.359671] Lustre: Skipped 1 previous similar message [ 1842.499588] Lustre: lustre-MDT0001: new disk, initializing [ 1842.710248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1842.736321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1847.666311] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1852.782929] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1859.140142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1859.360807] Lustre: lustre-OST0000: new disk, initializing [ 1859.364792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1859.371944] Lustre: 60783:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1861.172793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1861.201187] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1861.315390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1866.584824] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1878.799886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1878.981621] Lustre: lustre-OST0001: new disk, initializing [ 1878.989721] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1879.004895] Lustre: 61652:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1880.853406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1880.866523] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1880.967936] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1886.855885] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1897.138592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1900.773577] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1910.926567] Lustre: *** cfs_fail_loc=1603, val=0*** [ 1914.959422] Lustre: *** cfs_fail_loc=1604, val=0*** [ 1914.961159] Lustre: Skipped 19 previous similar messages [ 1920.633536] Lustre: *** cfs_fail_loc=1601, val=2*** [ 1920.640817] Lustre: Skipped 17 previous similar messages [ 1937.525780] Lustre: Failing over lustre-MDT0000 [ 1937.775728] Lustre: server umount lustre-MDT0000 complete [ 1939.939541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1948.180351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1948.313578] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1948.324362] LustreError: Skipped 3 previous similar messages [ 1948.585799] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1948.591456] Lustre: Skipped 1 previous similar message [ 1953.202859] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1953.774486] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1953.783555] Lustre: Skipped 1 previous similar message [ 1953.797753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1953.810674] Lustre: Skipped 7 previous similar messages [ 1953.860831] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1953.872068] Lustre: Skipped 1 previous similar message [ 1953.905235] Lustre: *** cfs_fail_loc=160b, val=2*** [ 1953.911993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:65) [ 1953.915505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:65) [ 1962.020457] Lustre: Failing over lustre-MDT0000 [ 1962.457499] Lustre: server umount lustre-MDT0000 complete [ 1972.920934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 1973.579904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1973.584387] Lustre: Skipped 8 previous similar messages [ 1978.968667] Lustre: *** cfs_fail_loc=160b, val=2*** [ 1978.987217] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:97) [ 1978.990056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:97) [ 1979.860307] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1987.289723] Lustre: Failing over lustre-MDT0000 [ 1987.662052] Lustre: server umount lustre-MDT0000 complete [ 1999.340759] LustreError: 59160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1999.370631] LustreError: 59160:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 52 previous similar messages [ 1999.495165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2005.586452] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:43 to 0x2c0000401:129) [ 2005.586693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:43 to 0x280000401:129) [ 2005.604988] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2012.149169] Lustre: *** cfs_fail_loc=1602, val=2*** [ 2026.573596] Lustre: DEBUG MARKER: == sanity-lfsck test 9a: LFSCK speed control (1) ========= 01:10:23 (1787461823) [ 2040.420918] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2063.593282] Lustre: 68612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2087.356819] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2091.765481] Lustre: 69748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2100.408882] Lustre: 59157:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 2100.416197] Lustre: 59157:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 2833 previous similar messages [ 2100.422218] Lustre: 59157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 2100.430661] Lustre: 59157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 2832 previous similar messages [ 2100.440061] Lustre: 59157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 2100.446912] Lustre: 59157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 2832 previous similar messages [ 2100.451110] Lustre: 59157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 2100.455601] Lustre: 59157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 2833 previous similar messages [ 2100.458853] Lustre: 59157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 2100.462074] Lustre: 59157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 2832 previous similar messages [ 2100.465737] Lustre: 59157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 2100.471282] Lustre: 59157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 2833 previous similar messages [ 2205.558576] Lustre: DEBUG MARKER: == sanity-lfsck test 9b: LFSCK speed control (2) ========= 01:13:21 (1787462001) [ 2257.167116] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2257.172106] Lustre: Skipped 4 previous similar messages [ 2284.667634] Lustre: *** cfs_fail_loc=160c, val=0*** [ 2284.672434] Lustre: Skipped 7 previous similar messages [ 2324.366619] Lustre: DEBUG MARKER: == sanity-lfsck test 10: System is available during LFSCK scanning ========================================================== 01:15:19 (1787462119) [ 2374.289828] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2375.309417] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2375.312503] Lustre: Skipped 63 previous similar messages [ 2377.329312] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2377.336064] Lustre: Skipped 109 previous similar messages [ 2381.331257] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2381.333216] Lustre: Skipped 205 previous similar messages [ 2389.333714] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2389.335883] Lustre: Skipped 341 previous similar messages [ 2405.347606] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2405.350687] Lustre: Skipped 791 previous similar messages [ 2437.414531] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2437.419512] Lustre: Skipped 1637 previous similar messages [ 2445.970327] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2445.974372] Lustre: Skipped 2599 previous similar messages [ 2690.905417] Lustre: DEBUG MARKER: == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ========================================================== 01:21:26 (1787462486) [ 2741.409240] Lustre: 62594:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 320, rollback = 2 [ 2741.420668] Lustre: 62594:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 36831 previous similar messages [ 2741.432424] Lustre: 62594:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 1/4/0 [ 2741.440035] Lustre: 62594:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2741.452509] Lustre: 62594:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 5/5/0, xattr_set: 7/320/0 [ 2741.464208] Lustre: 62594:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2741.477525] Lustre: 62594:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 6/46/0, punch: 0/0/0, quota 1/3/0 [ 2741.493968] Lustre: 62594:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2741.505380] Lustre: 62594:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 2/5/0 [ 2741.514639] Lustre: 62594:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2741.533419] Lustre: 62594:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1 [ 2741.549143] Lustre: 62594:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 36831 previous similar messages [ 2850.272148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2850.278856] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2850.283721] LustreError: Skipped 2 previous similar messages [ 2850.284699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2850.304087] Lustre: Skipped 18 previous similar messages [ 2852.014301] LustreError: 73594:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff98bb851e6800) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2852.130921] Lustre: server umount lustre-MDT0000 complete [ 2855.400043] LustreError: 70174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2856.639519] LustreError: 59143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462653 with bad export cookie 259145768625686090 [ 2856.640341] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2856.654574] LustreError: 59143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2856.661331] LustreError: Skipped 2 previous similar messages [ 2857.054310] Lustre: server umount lustre-MDT0001 complete [ 2871.670759] Lustre: server umount lustre-OST0000 complete [ 2885.301442] Lustre: server umount lustre-OST0001 complete [ 2892.587726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 2901.865965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 2917.471434] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2922.591289] LustreError: 74996:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 2948.319245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2948.334495] Lustre: Skipped 1 previous similar message [ 2954.791647] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2958.638943] Lustre: 75581:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 2958.666557] Lustre: *** cfs_fail_loc=160e, val=3*** [ 2961.700994] Lustre: 75581:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 2970.957205] Lustre: DEBUG MARKER: == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 01:26:07 (1787462767) [ 2986.412080] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2997.868313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.686362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3271 to 0x280000401:3297) [ 3005.269678] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3016.201198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3021.977451] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3026.174438] Lustre: 78245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3043.258679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3048.956357] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3208 to 0x2c0000401:3233) [ 3051.417268] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3059.862242] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3065.015280] Lustre: 79746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3069.930881] Lustre: *** cfs_fail_loc=160d, val=0*** [ 3076.510902] Lustre: Failing over lustre-OST0000 [ 3076.627973] Lustre: server umount lustre-OST0000 complete [ 3078.114120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3078.126646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.164411] Lustre: Skipped 2 previous similar messages [ 3090.245477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3090.544254] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3090.560334] Lustre: Skipped 2 previous similar messages [ 3092.523185] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3092.535641] Lustre: Skipped 2 previous similar messages [ 3092.574992] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3092.576260] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3092.582989] Lustre: *** cfs_fail_loc=215, val=0*** [ 3092.586743] Lustre: Skipped 2 previous similar messages [ 3092.628366] Lustre: Skipped 11 previous similar messages [ 3098.098145] Lustre: *** cfs_fail_loc=215, val=0*** [ 3098.105457] Lustre: Skipped 3 previous similar messages [ 3099.132516] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3103.199548] Lustre: *** cfs_fail_loc=215, val=0*** [ 3103.609857] Lustre: 81145:0:(ofd_dev.c:561:ofd_lfsck_out_notify()) lustre-OST0000: Found crashed LAST_ID, deny creating new OST-object on the device until the LAST_ID rebuilt successfully. [ 3103.634764] Lustre: 81145:0:(ofd_dev.c:573:ofd_lfsck_out_notify()) lustre-OST0000: Rebuilt crashed LAST_ID files successfully. [ 3107.232240] Lustre: Failing over lustre-OST0000 [ 3107.389272] Lustre: server umount lustre-OST0000 complete [ 3119.132591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3121.275774] Lustre: *** cfs_fail_loc=215, val=0*** [ 3121.283872] Lustre: Skipped 3 previous similar messages [ 3126.754729] Lustre: *** cfs_fail_loc=215, val=0*** [ 3126.763266] Lustre: Skipped 2 previous similar messages [ 3128.543770] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3134.432758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3134.440844] Lustre: Skipped 3 previous similar messages [ 3140.067948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3140.077168] Lustre: Skipped 6 previous similar messages [ 3140.859556] Lustre: server umount lustre-MDT0000 complete [ 3145.467375] LustreError: 75003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462942 with bad export cookie 259145768627259998 [ 3145.478421] LustreError: 75003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3146.089710] Lustre: server umount lustre-MDT0001 complete [ 3160.693718] Lustre: server umount lustre-OST0000 complete [ 3174.038614] Lustre: server umount lustre-OST0001 complete [ 3183.065351] Lustre: DEBUG MARKER: == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ========================================================== 01:29:39 (1787462979) [ 3197.965543] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3208.757478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3215.027596] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3224.974258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3230.641670] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3234.147714] Lustre: 85532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3240.973520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3249.008402] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3254.511089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3362 to 0x280000401:3393) [ 3258.248992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3264.001196] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3208 to 0x2c0000401:3265) [ 3267.194115] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3275.309509] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3280.353559] Lustre: 87403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3316.095629] Lustre: DEBUG MARKER: == sanity-lfsck test 12b: auto detect Lustre device ====== 01:31:52 (1787463112) [ 3331.270469] Lustre: DEBUG MARKER: == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ========================================================== 01:32:07 (1787463127) [ 3332.682879] Lustre: *** cfs_fail_loc=160f, val=0*** [ 3344.088174] Lustre: DEBUG MARKER: == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ========================================================== 01:32:20 (1787463140) [ 3344.386981] Lustre: 85602:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 3344.392334] Lustre: 85602:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1010 previous similar messages [ 3344.396845] Lustre: 85602:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 3344.399940] Lustre: 85602:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1010 previous similar messages [ 3344.406753] Lustre: 85602:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 3344.414191] Lustre: 85602:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1010 previous similar messages [ 3344.420216] Lustre: 85602:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/1 [ 3344.424615] Lustre: 85602:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1010 previous similar messages [ 3344.429750] Lustre: 85602:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 3344.437719] Lustre: 85602:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1010 previous similar messages [ 3344.443594] Lustre: 85602:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 3344.447649] Lustre: 85602:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1010 previous similar messages [ 3348.714699] Lustre: *** cfs_fail_loc=1610, val=0*** [ 3348.726204] Lustre: Skipped 7 previous similar messages [ 3402.215882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.219507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3402.239283] Lustre: Skipped 9 previous similar messages [ 3402.259956] Lustre: Skipped 3 previous similar messages [ 3406.876693] Lustre: server umount lustre-MDT0000 complete [ 3411.038422] LustreError: 85534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463208 with bad export cookie 259145768627268454 [ 3411.056581] LustreError: 85534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3411.351134] Lustre: server umount lustre-MDT0001 complete [ 3426.052566] Lustre: server umount lustre-OST0000 complete [ 3428.834421] Lustre: 16441:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787463209/real 1787463209] req@ffff98ba8c16c380 x1874288005200000/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787463225 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3429.733313] Lustre: server umount lustre-OST0001 complete [ 3449.653750] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3460.530133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3460.904842] LustreError: 92130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3460.924334] LustreError: 92130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 53 previous similar messages [ 3465.216298] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3472.618448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3476.835521] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3479.811673] Lustre: 93270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3485.598941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3486.970626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3554 to 0x280000401:3585) [ 3493.748470] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3497.462923] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:161) [ 3506.084474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3511.784045] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:161) [ 3511.785089] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3361) [ 3514.503693] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3522.571761] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3526.254773] Lustre: 95141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3533.394745] Lustre: DEBUG MARKER: == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ========================================================== 01:35:29 (1787463329) [ 3538.800626] Lustre: *** cfs_fail_loc=1610, val=0*** [ 3538.801836] Lustre: Skipped 63 previous similar messages [ 3562.980697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3562.993139] Lustre: Skipped 3 previous similar messages [ 3567.541555] Lustre: server umount lustre-MDT0000 complete [ 3571.688161] LustreError: 92112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463368 with bad export cookie 259145768627296853 [ 3571.710415] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3571.712149] LustreError: 92112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3571.729057] LustreError: Skipped 2 previous similar messages [ 3572.503457] Lustre: server umount lustre-MDT0001 complete [ 3587.538716] Lustre: server umount lustre-OST0000 complete [ 3589.343262] Lustre: 16439:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787463370/real 1787463370] req@ffff98bbb7457b80 x1874288005328256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787463386 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3592.116505] Lustre: server umount lustre-OST0001 complete [ 3610.459491] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3621.893857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3622.542693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3622.551411] Lustre: Skipped 13 previous similar messages [ 3628.112324] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3637.977242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3643.712090] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3647.136361] Lustre: 99176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3654.424553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3662.460677] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3662.957296] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:116 to 0x280000400:193) [ 3667.059398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3682 to 0x280000401:3713) [ 3672.115680] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3677.691624] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3317 to 0x2c0000401:3393) [ 3677.695833] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:116 to 0x2c0000400:193) [ 3679.341465] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3687.584552] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3698.346401] Lustre: DEBUG MARKER: == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ========================================================== 01:38:14 (1787463494) [ 3701.973712] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3701.975456] Lustre: Skipped 63 previous similar messages [ 3702.114406] Lustre: *** cfs_fail_loc=1611, val=0*** [ 3714.150393] Lustre: DEBUG MARKER: == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ========================================================== 01:38:30 (1787463510) [ 3716.945854] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3717.029348] Lustre: *** cfs_fail_loc=1612, val=0*** [ 3717.034199] Lustre: Skipped 2 previous similar messages [ 3729.135339] Lustre: DEBUG MARKER: == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ========================================================== 01:38:45 (1787463525) [ 3731.285762] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_15c MDS newer than 2.7.55, LU-6475 [ 3733.599581] Lustre: DEBUG MARKER: == sanity-lfsck test 15d: LFSCK don't crash upon dir migration failure ========================================================== 01:38:49 (1787463529) [ 3740.304963] Lustre: *** cfs_fail_loc=1709, val=0*** [ 3740.312507] LustreError: 98044:0:(mdt_reint.c:2644:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240002340:0x6c:0x0]/f20 failed: rc = -5 [ 3810.787385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.790527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3810.802514] Lustre: Skipped 10 previous similar messages [ 3810.813092] Lustre: Skipped 3 previous similar messages [ 3816.935995] Lustre: server umount lustre-MDT0000 complete [ 3827.107238] LustreError: 98017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463624 with bad export cookie 259145768627311581 [ 3827.131550] LustreError: 98017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3827.537442] Lustre: server umount lustre-MDT0001 complete [ 3847.175621] Lustre: server umount lustre-OST0000 complete [ 3847.647674] Lustre: 16441:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787463628/real 1787463628] req@ffff98ba93077480 x1874288005969280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787463644 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3852.769827] Lustre: 16442:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787463633/real 1787463633] req@ffff98ba825b7480 x1874288005969792/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787463649 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3852.824619] Lustre: 16442:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3857.908108] Lustre: 16441:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787463639/real 1787463639] req@ffff98ba825b5180 x1874288005970432/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787463655 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3857.960965] Lustre: 16441:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3859.583696] Lustre: server umount lustre-OST0001 complete [ 3883.800204] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3887.997724] Key type lgssc unregistered [ 3888.406468] LNet: 104896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3888.413397] LNetError: 104896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3888.465147] LNet: Removed LNI 192.168.201.118@tcp [ 3889.615182] Key type .llcrypt unregistered [ 3889.617936] Key type ._llcrypt unregistered [ 3920.662913] Key type ._llcrypt registered [ 3920.678734] Key type .llcrypt registered [ 3920.856191] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3937.113533] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3939.084699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3939.101793] alg: No test for adler32 (adler32-zlib) [ 3940.211911] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 3940.593434] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3942.311189] Key type lgssc registered [ 3943.552068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3999.214035] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4016.707439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4016.779803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4018.271640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4018.305383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4018.420709] Lustre: lustre-MDT0000: new disk, initializing [ 4018.582224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4018.603116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4024.275872] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4045.209788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4045.411914] Lustre: 109357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4045.516222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4045.520668] Lustre: Skipped 1 previous similar message [ 4045.788816] Lustre: lustre-MDT0001: new disk, initializing [ 4046.154789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4046.247937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4046.271739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4053.603823] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4060.092366] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4071.841228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4072.207050] Lustre: lustre-OST0000: new disk, initializing [ 4072.215826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4072.227248] Lustre: 111298:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4072.354946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4080.166169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4080.180638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4080.332830] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4080.342112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4097.062495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4097.279786] Lustre: lustre-OST0001: new disk, initializing [ 4097.285467] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4097.311411] Lustre: 112323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4097.429986] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4104.816777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4104.841431] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4104.918541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4105.487366] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4117.893366] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4124.166577] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4130.165801] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 01:45:26 (1787463926) === [ 4138.719179] Lustre: DEBUG MARKER: == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner ========================================================== 01:45:34 (1787463934) [ 4139.513243] Lustre: 109365:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 4139.545996] Lustre: 109365:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4139.555987] Lustre: 109365:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4139.568642] Lustre: 109365:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 4139.583951] Lustre: 109365:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 4139.599765] Lustre: 109365:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4140.044773] Lustre: 109365:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 4140.062589] Lustre: 109365:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 6 previous similar messages [ 4140.079560] Lustre: 109365:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 4140.092393] Lustre: 109365:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 4140.098667] Lustre: 109365:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4140.106076] Lustre: 109365:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 4140.120772] Lustre: 109365:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 4140.125676] Lustre: 109365:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 4140.131880] Lustre: 109365:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 4140.136960] Lustre: 109365:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 4140.143313] Lustre: 109365:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4140.147483] Lustre: 109365:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 6 previous similar messages [ 4141.045350] Lustre: 109364:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 4141.055594] Lustre: 109364:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 74 previous similar messages [ 4141.105719] Lustre: 112331:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 4141.116991] Lustre: 112331:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 77 previous similar messages [ 4141.124341] Lustre: 112331:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4141.137017] Lustre: 112331:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 77 previous similar messages [ 4141.147385] Lustre: 112331:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 7/369/0 [ 4141.159556] Lustre: 112331:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 77 previous similar messages [ 4141.168792] Lustre: 112331:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 4141.191519] Lustre: 112331:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 77 previous similar messages [ 4141.206801] Lustre: 112331:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4141.228559] Lustre: 112331:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 77 previous similar messages [ 4143.995518] Lustre: *** cfs_fail_loc=1613, val=0*** [ 4158.648272] Lustre: DEBUG MARKER: == sanity-lfsck test 17: LFSCK can repair multiple references ========================================================== 01:45:54 (1787463954) [ 4160.126759] Lustre: 109366:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 257 < left 278, rollback = 2 [ 4160.139068] Lustre: 109366:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 227 previous similar messages [ 4160.167623] Lustre: 109366:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 4160.177194] Lustre: 109366:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 224 previous similar messages [ 4160.183198] Lustre: 109366:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4160.192093] Lustre: 109366:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 224 previous similar messages [ 4160.200455] Lustre: 109366:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 4160.208394] Lustre: 109366:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 224 previous similar messages [ 4160.213275] Lustre: 109366:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 4160.232901] Lustre: 109366:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 224 previous similar messages [ 4160.241566] Lustre: 109366:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4160.255242] Lustre: 109366:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 224 previous similar messages [ 4161.861851] Lustre: *** cfs_fail_loc=1614, val=0*** [ 4162.403398] Lustre: *** cfs_fail_loc=1614, val=103*** [ 4162.407193] Lustre: Skipped 1 previous similar message [ 4166.926152] Lustre: 111289:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 275, rollback = 2 [ 4166.934060] Lustre: 111289:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 11 previous similar messages [ 4166.941651] Lustre: 111289:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4166.949847] Lustre: 111289:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4166.960646] Lustre: 111289:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 3/275/0 [ 4166.971591] Lustre: 111289:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4166.981055] Lustre: 111289:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 1/4/0, quota 7/225/0 [ 4166.995502] Lustre: 111289:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4167.003708] Lustre: 111289:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 4167.014567] Lustre: 111289:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4167.034262] Lustre: 111289:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4167.050617] Lustre: 111289:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages [ 4176.021370] Lustre: DEBUG MARKER: == sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) ========================================================== 01:46:11 (1787463971) [ 4176.512811] Lustre: 112331:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 4176.519342] Lustre: 112331:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 4176.524571] Lustre: 112331:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4176.533281] Lustre: 112331:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4176.543382] Lustre: 112331:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4176.551831] Lustre: 112331:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4176.563287] Lustre: 112331:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 4176.574324] Lustre: 112331:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4176.583268] Lustre: 112331:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 4176.601985] Lustre: 112331:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4176.610334] Lustre: 112331:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4176.619885] Lustre: 112331:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4179.530140] Lustre: *** cfs_fail_loc=1615, val=0*** [ 4179.534279] Lustre: Skipped 1 previous similar message [ 4180.907776] Lustre: *** cfs_fail_loc=1615, val=0*** [ 4180.911355] Lustre: Skipped 1 previous similar message [ 4201.118738] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_18b skipping excluded test 18b [ 4202.734531] Lustre: DEBUG MARKER: == sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) ========================================================== 01:46:39 (1787463999) [ 4203.147350] Lustre: 112330:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 4203.153657] Lustre: 112330:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 21 previous similar messages [ 4203.158879] Lustre: 112330:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 4203.163455] Lustre: 112330:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4203.169595] Lustre: 112330:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4203.175197] Lustre: 112330:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4203.179632] Lustre: 112330:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 4203.185039] Lustre: 112330:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4203.190830] Lustre: 112330:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/1, delete: 0/0/0 [ 4203.195871] Lustre: 112330:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4203.200761] Lustre: 112330:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4203.205622] Lustre: 112330:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4206.112588] Lustre: *** cfs_fail_loc=1617, val=0*** [ 4206.188892] Lustre: *** cfs_fail_loc=1617, val=0*** [ 4208.672486] Lustre: *** cfs_fail_loc=1616, val=0*** [ 4208.678362] Lustre: Skipped 3 previous similar messages [ 4221.557861] Lustre: DEBUG MARKER: sanity-lfsck test_18c: @@@@@@ FAIL: (4) Expect 4 fixed on mds1, but got: 8 [ 4228.340543] Lustre: DEBUG MARKER: == sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) ========================================================== 01:47:04 (1787464024) [ 4230.449392] Lustre: *** cfs_fail_loc=1618, val=0*** [ 4230.453979] Lustre: Skipped 5 previous similar messages [ 4265.954089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4265.958109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4265.976979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4269.025581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4269.027644] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4269.037574] Lustre: Skipped 2 previous similar messages [ 4269.055601] Lustre: Skipped 2 previous similar messages [ 4272.479558] Lustre: server umount lustre-MDT0000 complete [ 4276.388644] LustreError: 110307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787464073 with bad export cookie 12680239035058574421 [ 4276.394768] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4276.403824] LustreError: 110307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4276.733529] Lustre: server umount lustre-MDT0001 complete [ 4291.014916] Lustre: server umount lustre-OST0000 complete [ 4295.135695] Lustre: 106515:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787464076/real 1787464076] req@ffff98ba948a8000 x1874291575708672/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787464092 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4295.176054] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4295.801489] Lustre: server umount lustre-OST0001 complete [ 4316.794197] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4329.767397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4330.504950] LustreError: 118040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4330.534774] LustreError: 118040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4330.661585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4336.099092] LustreError: 118041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4336.248553] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4340.198449] LustreError: 118040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4345.312984] LustreError: 118041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4347.491707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4347.923344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4353.074329] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4356.036335] Lustre: 119181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4363.077979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4369.796596] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4372.579861] LustreError: 119537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4372.622710] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:33) [ 4375.666501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:116 to 0x280000401:161) [ 4378.457573] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4378.683843] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4378.686787] Lustre: Skipped 1 previous similar message [ 4383.728182] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 4383.735751] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:33) [ 4386.366659] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4396.084923] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4400.281406] Lustre: 121053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4416.561289] Lustre: DEBUG MARKER: == sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) ========================================================== 01:50:12 (1787464212) [ 4417.105890] Lustre: 120362:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 251 < left 278, rollback = 2 [ 4417.123444] Lustre: 120362:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 54 previous similar messages [ 4417.132931] Lustre: 120362:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4417.140725] Lustre: 120362:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 54 previous similar messages [ 4417.164451] Lustre: 120362:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4417.177564] Lustre: 120362:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 54 previous similar messages [ 4417.186246] Lustre: 120362:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 4417.192407] Lustre: 120362:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 54 previous similar messages [ 4417.222793] Lustre: 120362:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 4417.236785] Lustre: 120362:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 54 previous similar messages [ 4417.248972] Lustre: 120362:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4417.259647] Lustre: 120362:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 54 previous similar messages [ 4419.326755] Lustre: *** cfs_fail_loc=1618, val=0*** [ 4419.328818] Lustre: Skipped 3 previous similar messages [ 4455.395171] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4455.400896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4455.420105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4455.430140] Lustre: Skipped 3 previous similar messages [ 4460.512331] LustreError: 118041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4460.526659] LustreError: 118041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4462.025587] Lustre: server umount lustre-MDT0000 complete [ 4465.451202] LustreError: 118020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787464262 with bad export cookie 12680239035058589926 [ 4465.457557] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4465.463171] LustreError: 118020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4465.632153] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4465.663648] Lustre: Skipped 3 previous similar messages [ 4465.668297] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4465.681196] Lustre: Skipped 1 previous similar message [ 4466.017373] Lustre: server umount lustre-MDT0001 complete [ 4472.892481] Lustre: server umount lustre-OST0000 complete [ 4478.661124] Lustre: server umount lustre-OST0001 complete [ 4499.446388] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4514.083248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4514.891370] LustreError: 123623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4514.910198] LustreError: 123623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4514.978384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4519.208546] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4530.417045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4530.658497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4537.302678] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4540.924503] Lustre: 124764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4548.849582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4555.371833] LustreError: 125116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4555.377662] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4 to 0x280000400:65) [ 4555.391747] LustreError: 125116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4557.494883] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4560.514822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4569.908256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4575.736587] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4 to 0x2c0000401:65) [ 4575.744266] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 4580.060680] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4590.237513] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4595.428033] Lustre: 126638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4601.269040] Lustre: 124557:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 252 < left 260, rollback = 2 [ 4601.276965] Lustre: 124557:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 21 previous similar messages [ 4601.281612] Lustre: 124557:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4601.286286] Lustre: 124557:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4601.294249] Lustre: 124557:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/260/0 [ 4601.305542] Lustre: 124557:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4601.319108] Lustre: 124557:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/9/0, punch: 0/0/0, quota 4/150/2 [ 4601.327196] Lustre: 124557:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4601.343453] Lustre: 124557:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 1/17/2, delete: 0/0/0 [ 4601.353383] Lustre: 124557:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4601.365500] Lustre: 124557:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4601.372505] Lustre: *** cfs_fail_loc=1602, val=10*** [ 4601.373785] Lustre: 124557:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages [ 4631.731542] Lustre: DEBUG MARKER: == sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ========================================================== 01:53:48 (1787464428) [ 4635.375142] Lustre: *** cfs_fail_loc=1616, val=0*** [ 4635.377529] Lustre: Skipped 3 previous similar messages [ 4643.801515] Lustre: *** cfs_fail_loc=161c, val=0*** [ 4671.175521] Lustre: DEBUG MARKER: == sanity-lfsck test 18g: Find out orphan OST-object and repair it (7) ========================================================== 01:54:26 (1787464466) [ 4674.178881] Lustre: *** cfs_fail_loc=162e, val=0*** [ 4674.187182] Lustre: *** cfs_fail_loc=162e, val=0*** [ 4674.195025] Lustre: Skipped 7 previous similar messages [ 4692.220929] Lustre: DEBUG MARKER: == sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ========================================================== 01:54:48 (1787464488) [ 4720.007477] Lustre: DEBUG MARKER: == sanity-lfsck test 19a: OST-object inconsistency self detect ========================================================== 01:55:16 (1787464516) [ 4735.086284] Lustre: DEBUG MARKER: == sanity-lfsck test 19b: OST-object inconsistency self repair ========================================================== 01:55:30 (1787464530) [ 4735.613898] Lustre: 123620:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 4735.627301] Lustre: 123620:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 94 previous similar messages [ 4735.640554] Lustre: 123620:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 4735.647862] Lustre: 123620:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 94 previous similar messages [ 4735.662048] Lustre: 123620:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 4735.669059] Lustre: 123620:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 94 previous similar messages [ 4735.678577] Lustre: 123620:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 4735.686790] Lustre: 123620:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 94 previous similar messages [ 4735.699910] Lustre: 123620:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 4735.710699] Lustre: 123620:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 94 previous similar messages [ 4735.718110] Lustre: 123620:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 4735.727396] Lustre: 123620:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 94 previous similar messages [ 4740.097777] Lustre: *** cfs_fail_loc=1611, val=0*** [ 4740.124325] Lustre: *** cfs_fail_loc=1611, val=0*** [ 4740.129184] Lustre: Skipped 3 previous similar messages [ 4746.356227] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.18@tcp inode [0x2000013a1:0x1f:0x0] object 0x280000401:203 extent [0-4095], client returned csum 0 (type 4), server csum 703fb3ad (type 4) [ 4747.399506] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.18@tcp inode [0x2000013a1:0x20:0x0] object 0x280000401:204 extent [0-4095], client returned csum 0 (type 4), server csum 7cbb935c (type 4) [ 4758.162701] Lustre: DEBUG MARKER: == sanity-lfsck test 20a: Handle the orphan with dummy LOV EA slot properly ========================================================== 01:55:53 (1787464553) [ 4761.769503] Lustre: *** cfs_fail_loc=1616, val=0*** [ 4761.775861] Lustre: Skipped 3 previous similar messages [ 4791.159870] Lustre: DEBUG MARKER: == sanity-lfsck test 20b: Handle the orphan with dummy LOV EA slot properly - PFL case ========================================================== 01:56:27 (1787464587) [ 4800.520181] Lustre: DEBUG MARKER: == sanity-lfsck test 21: run all LFSCK components by default ========================================================== 01:56:36 (1787464596) [ 4817.368441] Lustre: DEBUG MARKER: == sanity-lfsck test 22a: LFSCK can repair unmatched pairs (1) ========================================================== 01:56:53 (1787464613) [ 4820.995334] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4821.004520] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4821.014897] Lustre: Skipped 1 previous similar message [ 4835.849800] Lustre: DEBUG MARKER: == sanity-lfsck test 22b: LFSCK can repair unmatched pairs (2) ========================================================== 01:57:12 (1787464632) [ 4837.778658] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4837.792499] Lustre: *** cfs_fail_loc=161e, val=0*** [ 4850.945983] Lustre: DEBUG MARKER: == sanity-lfsck test 23a: LFSCK can repair dangling name entry (1) ========================================================== 01:57:27 (1787464647) [ 4852.925430] Lustre: *** cfs_fail_loc=1620, val=0*** [ 4870.059388] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_23b skipping excluded test 23b [ 4872.089540] Lustre: DEBUG MARKER: == sanity-lfsck test 23c: LFSCK can repair dangling name entry (3) ========================================================== 01:57:48 (1787464668) [ 4879.493764] Lustre: *** cfs_fail_loc=1621, val=127*** [ 4879.501044] Lustre: Skipped 1 previous similar message [ 4883.207296] Lustre: *** cfs_fail_loc=1602, val=10*** [ 4907.672928] Lustre: DEBUG MARKER: == sanity-lfsck test 23d: LFSCK can repair a dangling name entry to a remote object ========================================================== 01:58:23 (1787464703) [ 4910.428440] Lustre: Failing over lustre-MDT0000 [ 4910.978642] Lustre: server umount lustre-MDT0000 complete [ 4911.589512] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4911.606581] Lustre: Skipped 1 previous similar message [ 4911.612388] LustreError: 123619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4911.638866] LustreError: 123619:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4923.173725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4923.370783] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4923.712861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4923.719488] Lustre: Skipped 3 previous similar messages [ 4923.760773] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4927.606831] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4929.003460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4929.020030] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4929.091805] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:130 to 0x2c0000401:161) [ 4929.104435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:265 to 0x280000401:289) [ 4929.401520] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4932.015413] LustreError: 123617:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0000: [0x2000013a3:0x82:0x0] doesn't exist!: rc = -14 [ 4945.266887] Lustre: DEBUG MARKER: == sanity-lfsck test 24: LFSCK can repair multiple-referenced name entry ========================================================== 01:59:01 (1787464741) [ 4947.491224] Lustre: *** cfs_fail_loc=1622, val=0*** [ 4947.724493] Lustre: *** cfs_fail_loc=1622, val=0*** [ 4947.729353] Lustre: Skipped 1 previous similar message [ 4962.330041] Lustre: DEBUG MARKER: == sanity-lfsck test 25: LFSCK can repair bad file type in the name entry ========================================================== 01:59:18 (1787464758) [ 4964.477273] Lustre: *** cfs_fail_loc=1623, val=0*** [ 4978.391301] Lustre: DEBUG MARKER: == sanity-lfsck test 26a: LFSCK can add the missing local name entry back to the namespace ========================================================== 01:59:34 (1787464774) [ 4980.188354] Lustre: *** cfs_fail_loc=1624, val=0*** [ 4993.592738] Lustre: DEBUG MARKER: == sanity-lfsck test 26b: LFSCK can add the missing remote name entry back to the namespace ========================================================== 01:59:49 (1787464789) [ 4994.044741] Lustre: 123617:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 257 < left 320, rollback = 2 [ 4994.060230] Lustre: 123617:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 476 previous similar messages [ 4994.074621] Lustre: 123617:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 1/4/0 [ 4994.088181] Lustre: 123617:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 476 previous similar messages [ 4994.102501] Lustre: 123617:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 5/5/1, xattr_set: 7/320/0 [ 4994.108081] Lustre: 123617:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 476 previous similar messages [ 4994.121359] Lustre: 123617:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 6/46/0, punch: 0/0/0, quota 1/3/0 [ 4994.130208] Lustre: 123617:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 476 previous similar messages [ 4994.146748] Lustre: 123617:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 2/5/1 [ 4994.151144] Lustre: 123617:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 476 previous similar messages [ 4994.164134] Lustre: 123617:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1 [ 4994.173685] Lustre: 123617:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 476 previous similar messages [ 5008.842385] Lustre: DEBUG MARKER: == sanity-lfsck test 27a: LFSCK can recreate the lost local parent directory as orphan ========================================================== 02:00:04 (1787464804) [ 5010.742553] Lustre: *** cfs_fail_loc=1624, val=0*** [ 5010.753227] Lustre: Skipped 1 previous similar message [ 5025.867725] Lustre: DEBUG MARKER: == sanity-lfsck test 27b: LFSCK can recreate the lost remote parent directory as orphan ========================================================== 02:00:21 (1787464821) [ 5042.854603] Lustre: DEBUG MARKER: == sanity-lfsck test 28: Skip the failed MDT(s) when handle orphan MDT-objects ========================================================== 02:00:38 (1787464838) [ 5045.719536] Lustre: *** cfs_fail_loc=1624, val=0*** [ 5045.721432] Lustre: Skipped 2 previous similar messages [ 5050.355078] Lustre: *** cfs_fail_loc=161c, val=0*** [ 5050.356882] Lustre: Skipped 1 previous similar message [ 5070.519535] Lustre: DEBUG MARKER: == sanity-lfsck test 29b: LFSCK can repair bad nlink count (2) ========================================================== 02:01:06 (1787464866) [ 5089.755562] Lustre: DEBUG MARKER: == sanity-lfsck test 29c: verify linkEA size limitation == 02:01:25 (1787464885) [ 5131.062941] Lustre: DEBUG MARKER: == sanity-lfsck test 29d: accessing non-existing inode shouldn't turn fs read-only (ldiskfs) ========================================================== 02:02:06 (1787464926) [ 5133.590403] Lustre: *** cfs_fail_loc=1626, val=0*** [ 5133.595979] Lustre: Skipped 2 previous similar messages [ 5135.417490] LustreError: 125139:0:(osd_handler.c:284:osd_idc_find_or_init()) lustre-MDT0000: cannot lookup FID [0x2000013a3:0x9e:0x0]: rc = -2 [ 5143.870876] Lustre: DEBUG MARKER: == sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ========================================================== 02:02:19 (1787464939) [ 5179.587456] Lustre: Failing over lustre-MDT0000 [ 5179.872956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5179.874595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5179.895629] Lustre: Skipped 5 previous similar messages [ 5180.292665] Lustre: server umount lustre-MDT0000 complete [ 5185.000246] LustreError: 123624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5185.024239] LustreError: 123624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 5195.463877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5195.643857] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5195.965099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5196.015921] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5200.561317] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5201.377980] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5201.400950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5201.426738] Lustre: Skipped 3 previous similar messages [ 5201.447696] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5201.497404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:295 to 0x280000401:321) [ 5201.501976] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:166 to 0x2c0000401:193) [ 5217.140766] Lustre: DEBUG MARKER: == sanity-lfsck test 31a: The LFSCK can find/repair the name entry with bad name hash (1) ========================================================== 02:03:32 (1787465012) [ 5237.088868] Lustre: DEBUG MARKER: == sanity-lfsck test 31b: The LFSCK can find/repair the name entry with bad name hash (2) ========================================================== 02:03:53 (1787465033) [ 5263.106474] Lustre: DEBUG MARKER: == sanity-lfsck test 31c: Re-generate the lost master LMV EA for striped directory ========================================================== 02:04:18 (1787465058) [ 5264.882554] Lustre: *** cfs_fail_loc=1629, val=0*** [ 5264.887622] Lustre: Skipped 5 previous similar messages [ 5294.060358] Lustre: DEBUG MARKER: == sanity-lfsck test 31d: Set broken striped directory (modified after broken) as read-only ========================================================== 02:04:50 (1787465090) [ 5303.348027] Lustre: Failing over lustre-MDT0000 [ 5303.796664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5303.808524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.839778] Lustre: Skipped 1 previous similar message [ 5304.005677] Lustre: server umount lustre-MDT0000 complete [ 5316.049602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5316.181735] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5316.439909] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5321.703495] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5321.703772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5321.726887] Lustre: Skipped 3 previous similar messages [ 5321.800175] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5321.865551] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:166 to 0x2c0000401:225) [ 5321.873249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:295 to 0x280000401:353) [ 5322.098867] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5335.195693] Lustre: Failing over lustre-MDT0000 [ 5335.713955] Lustre: server umount lustre-MDT0000 complete [ 5337.063256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5346.498188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5346.640126] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5347.221351] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5351.027231] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5352.451344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5352.458323] Lustre: Skipped 3 previous similar messages [ 5352.493854] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5352.576620] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:166 to 0x2c0000401:257) [ 5352.576979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5353.854311] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5367.274078] Lustre: DEBUG MARKER: == sanity-lfsck test 31e: Re-generate the lost slave LMV EA for striped directory (1) ========================================================== 02:06:02 (1787465162) [ 5381.891681] Lustre: DEBUG MARKER: == sanity-lfsck test 31f: Re-generate the lost slave LMV EA for striped directory (2) ========================================================== 02:06:18 (1787465178) [ 5398.551321] Lustre: DEBUG MARKER: == sanity-lfsck test 31g: Repair the corrupted slave LMV EA ========================================================== 02:06:34 (1787465194) [ 5441.440211] Lustre: DEBUG MARKER: == sanity-lfsck test 31h: Repair the corrupted shard's name entry ========================================================== 02:07:17 (1787465237) [ 5462.709761] Lustre: DEBUG MARKER: == sanity-lfsck test 32a: stop LFSCK when some OST failed ========================================================== 02:07:38 (1787465258) [ 5472.782825] LustreError: 148043:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5475.880124] LustreError: 148043:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d awake [ 5475.893498] LustreError: 148043:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5476.030637] Lustre: Failing over lustre-OST0000 [ 5476.136912] Lustre: server umount lustre-OST0000 complete [ 5477.345945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5477.358672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5477.381438] Lustre: Skipped 7 previous similar messages [ 5477.395276] LustreError: 125119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5477.415981] LustreError: 125119:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 31 previous similar messages [ 5478.983278] LustreError: 148043:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d awake [ 5479.000636] LustreError: 148043:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5479.655105] LustreError: 148043:0:(lfsck_layout.c:4529:lfsck_layout_assistant_handler_p1()) cfs_fail_timeout interrupted [ 5479.689740] LustreError: lustre-OST0000-osc-MDT0000: operation lfsck_notify to node 0@lo failed: rc = -107 [ 5493.685993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5494.062815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5494.075861] Lustre: Skipped 2 previous similar messages [ 5494.091836] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5495.205642] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5495.247078] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5495.252206] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5495.270719] Lustre: Skipped 3 previous similar messages [ 5502.122757] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5513.715996] Lustre: DEBUG MARKER: == sanity-lfsck test 32b: stop LFSCK when some MDT failed ========================================================== 02:08:29 (1787465309) [ 5533.279586] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5557.515817] Lustre: 150849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5587.688941] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5592.549716] Lustre: 151985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5595.034686] Lustre: 123620:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 5595.042736] Lustre: 123620:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 865 previous similar messages [ 5595.046464] Lustre: 123620:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 5595.049588] Lustre: 123620:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 865 previous similar messages [ 5595.054680] Lustre: 123620:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 5595.059173] Lustre: 123620:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 865 previous similar messages [ 5595.063322] Lustre: 123620:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 5595.066759] Lustre: 123620:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 865 previous similar messages [ 5595.070611] Lustre: 123620:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/2, delete: 0/0/0 [ 5595.075707] Lustre: 123620:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 865 previous similar messages [ 5595.081532] Lustre: 123620:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 5595.091434] Lustre: 123620:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 865 previous similar messages [ 5607.429512] LustreError: 152123:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d sleeping for 3000ms [ 5607.454786] LustreError: 152123:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) Skipped 1 previous similar message [ 5610.464729] LustreError: 152123:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d awake [ 5610.478473] LustreError: 152123:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) Skipped 1 previous similar message [ 5610.841527] Lustre: Failing over lustre-MDT0001 [ 5612.013799] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.021743] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5612.044387] Lustre: Skipped 3 previous similar messages [ 5612.069755] Lustre: Skipped 5 previous similar messages [ 5613.495471] LustreError: 152122:0:(lfsck_striped_dir.c:1762:lfsck_namespace_verify_stripe_slave()) cfs_fail_timeout id 162d awake [ 5614.399937] Lustre: server umount lustre-MDT0001 complete [ 5633.458105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 5633.994261] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5639.144872] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5639.148369] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5639.195771] Lustre: Skipped 1 previous similar message [ 5639.224701] Lustre: lustre-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5639.283284] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:97) [ 5639.305175] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:97) [ 5640.051837] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5651.352050] Lustre: DEBUG MARKER: == sanity-lfsck test 33: check LFSCK paramters =========== 02:10:47 (1787465447) [ 5672.463489] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5697.471191] Lustre: 154847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5733.681480] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5740.665668] Lustre: 155984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5770.802337] Lustre: DEBUG MARKER: == sanity-lfsck test 34: LFSCK can rebuild the lost agent object ========================================================== 02:12:46 (1787465566) [ 5773.580560] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_34 Only valid for ZFS backend [ 5776.181899] Lustre: DEBUG MARKER: == sanity-lfsck test 35: LFSCK can rebuild the lost agent entry ========================================================== 02:12:51 (1787465571) [ 5789.914868] Lustre: *** cfs_fail_loc=1631, val=0*** [ 5808.103882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5808.113410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5808.124372] Lustre: Skipped 1 previous similar message [ 5810.256285] Lustre: server umount lustre-MDT0000 complete [ 5815.517384] LustreError: 123605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787465612 with bad export cookie 12680239035058662859 [ 5815.520343] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5815.527167] LustreError: 123605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5815.982369] Lustre: server umount lustre-MDT0001 complete [ 5831.830808] Lustre: server umount lustre-OST0000 complete [ 5834.604387] Lustre: 106515:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787465615/real 1787465615] req@ffff98ba9302dc00 x1874291577310976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787465631 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5836.529806] Lustre: 106512:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787465617/real 1787465617] req@ffff98bbac156a00 x1874291577311360/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787465633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5837.266401] Lustre: server umount lustre-OST0001 complete [ 5858.649449] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5874.020119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5881.314531] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5893.104911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5899.558082] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5902.958943] Lustre: 159923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5911.670931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5918.386136] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:129) [ 5919.418695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:540 to 0x280000401:577) [ 5920.608391] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5931.899509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5937.667847] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:412 to 0x2c0000401:449) [ 5937.668141] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:69 to 0x2c0000400:129) [ 5940.696429] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5951.124763] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5955.779792] Lustre: 161793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5968.615225] Lustre: DEBUG MARKER: == sanity-lfsck test 36a: rebuild LOV EA for mirrored file (1) ========================================================== 02:16:04 (1787465764) [ 5970.365772] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36a needs >= 3 OSTs [ 5972.588215] Lustre: DEBUG MARKER: == sanity-lfsck test 36b: rebuild LOV EA for mirrored file (2) ========================================================== 02:16:08 (1787465768) [ 5974.871533] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36b needs >= 3 OSTs [ 5977.918729] Lustre: DEBUG MARKER: == sanity-lfsck test 36c: rebuild LOV EA for mirrored file (3) ========================================================== 02:16:13 (1787465773) [ 5980.079427] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_36c needs >= 3 OSTs [ 5982.341447] Lustre: DEBUG MARKER: == sanity-lfsck test 37: LFSCK must skip a ORPHAN ======== 02:16:18 (1787465778) [ 5996.069574] Lustre: DEBUG MARKER: == sanity-lfsck test 38: LFSCK does not break foreign file and reverse is also true ========================================================== 02:16:31 (1787465791) [ 6013.086816] Lustre: DEBUG MARKER: == sanity-lfsck test 39: LFSCK does not break foreign dir and reverse is also true ========================================================== 02:16:49 (1787465809) [ 6032.511646] Lustre: DEBUG MARKER: == sanity-lfsck test 40a: LFSCK correctly fixes lmm_oi in composite layout ========================================================== 02:17:08 (1787465828) [ 6055.920324] Lustre: DEBUG MARKER: == sanity-lfsck test 41: SEL support in LFSCK ============ 02:17:31 (1787465851) [ 6080.926021] Lustre: DEBUG MARKER: == sanity-lfsck test 42: LFSCK repairs inconsistent MDT-object/OST-object encryption flags ========================================================== 02:17:56 (1787465876) [ 6115.739671] Lustre: *** cfs_fail_loc=1632, val=0*** [ 6136.942807] Lustre: DEBUG MARKER: == sanity-lfsck test 43: LFSCK does not loop endlessly on iget failure in scanning-phase1 ========================================================== 02:18:52 (1787465932) [ 6140.846335] Lustre: Failing over lustre-MDT0001 [ 6141.401205] Lustre: server umount lustre-MDT0001 complete [ 6142.113415] LustreError: 160998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6142.150958] LustreError: 160998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 34 previous similar messages [ 6142.436371] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6142.458259] Lustre: Skipped 6 previous similar messages [ 6150.627374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6151.069845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6151.072857] Lustre: Skipped 5 previous similar messages [ 6151.116476] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6151.123624] Lustre: lustre-MDT0001: Aborting client recovery [ 6151.129161] LustreError: 165595:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0001: Aborting recovery [ 6151.134410] Lustre: 165619:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 6151.144472] Lustre: 165619:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client 4e0feaff-aa75-4905-8615-1c337699b1fe@ [ 6151.152282] Lustre: lustre-MDT0001: disconnecting 2 stale clients [ 6151.158080] Lustre: lustre-MDT0001-osd: cancel update llog [0x240000400:0x1:0x0] [ 6151.166648] Lustre: lustre-MDT0000-osp-MDT0001: cancel update llog [0x200000401:0x1:0x0] [ 6151.214503] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:161) [ 6151.214952] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:131 to 0x2c0000400:161) [ 6156.267992] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6156.277305] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6156.282988] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6156.303890] Lustre: Skipped 4 previous similar messages [ 6162.130069] Lustre: *** cfs_fail_loc=1e2, val=483*** [ 6169.514161] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid [ 6169.952904] Lustre: DEBUG MARKER: osp.lustre-MDT0001-os[pc]-MDT0000.*_server_uuid in FULL state after 0 sec [ 6179.701582] Lustre: DEBUG MARKER: == sanity-lfsck test 44: umount while lfsck is stopping == 02:19:35 (1787465975) [ 6191.414218] Lustre: *** cfs_fail_loc=1600, val=3*** [ 6194.852780] Lustre: Failing over lustre-MDT0000 [ 6195.158317] Lustre: server umount lustre-MDT0000 complete [ 6197.219400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6209.060388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6209.146862] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6209.381831] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6209.387262] Lustre: Skipped 2 previous similar messages [ 6213.756600] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6214.638515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6214.688287] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6214.731555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:621 to 0x280000401:641) [ 6214.732434] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:489 to 0x2c0000401:513) [ 6215.801870] Lustre: 160281:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 2: before 260 < left 274, rollback = 2 [ 6215.811899] Lustre: 160283:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 6215.830054] Lustre: 160281:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1625 previous similar messages [ 6215.830077] Lustre: 160281:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/274/0 [ 6215.830081] Lustre: 160281:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1623 previous similar messages [ 6215.830087] Lustre: 160281:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 7/225/0 [ 6215.830091] Lustre: 160281:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1624 previous similar messages [ 6215.830098] Lustre: 160281:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 6215.830102] Lustre: 160281:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1624 previous similar messages [ 6215.830108] Lustre: 160281:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 6215.830112] Lustre: 160281:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1624 previous similar messages [ 6215.964487] Lustre: 160283:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1627 previous similar messages [ 6216.982398] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6233.384347] Lustre: DEBUG MARKER: == sanity-lfsck test 45: LFSCK should fix UID/GID/PROJID of OST object ========================================================== 02:20:29 (1787466029) [ 6286.798913] Lustre: Failing over lustre-OST0000 [ 6286.816541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6286.831758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6286.840047] Lustre: Skipped 2 previous similar messages [ 6287.009376] Lustre: server umount lustre-OST0000 complete [ 6296.431543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6313.024173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6313.407037] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6315.212907] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6315.328355] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6315.337253] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6315.363698] Lustre: Skipped 3 previous similar messages [ 6322.422422] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6330.513351] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6330.773500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6336.323339] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6336.571354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6346.805420] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8bd4728a9000.ost_server_uuid 50 [ 6349.796818] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8bd4728a9000.ost_server_uuid in IDLE state after 0 sec [ 6450.144537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6450.157281] LustreError: Skipped 1 previous similar message [ 6450.163462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6451.857674] Lustre: server umount lustre-MDT0000 complete [ 6463.016491] LustreError: 158761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787466260 with bad export cookie 12680239035058745865 [ 6463.025141] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6463.045300] LustreError: 158761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6463.546468] Lustre: server umount lustre-MDT0001 complete [ 6483.423198] Lustre: 106514:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787466264/real 1787466264] req@ffff98ba8256e680 x1874291577776640/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787466280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6484.447192] Lustre: 106512:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787466265/real 1787466265] req@ffff98ba81c61f80 x1874291577777024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787466281 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6484.732333] Lustre: server umount lustre-OST0000 complete [ 6488.543298] Lustre: 106515:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787466269/real 1787466269] req@ffff98ba8af33480 x1874291577777280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787466285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6494.751413] Lustre: 106515:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787466275/real 1787466275] req@ffff98ba8af33800 x1874291577778048/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787466291 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6494.796853] Lustre: 106515:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6496.410203] Lustre: server umount lustre-OST0001 complete [ 6520.130708] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6523.964735] Key type lgssc unregistered [ 6524.384614] LNet: 175251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6524.400561] LNetError: 175251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6525.426841] LNet: Removed LNI 192.168.201.118@tcp [ 6526.637342] Key type .llcrypt unregistered [ 6526.640214] Key type ._llcrypt unregistered [ 6562.174935] Key type ._llcrypt registered [ 6562.176569] Key type .llcrypt registered [ 6562.348402] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6589.162064] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6592.170265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6592.508957] alg: No test for adler32 (adler32-zlib) [ 6594.257803] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 6594.834470] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6596.631171] Key type lgssc registered [ 6598.600400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6669.682361] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6689.151928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6689.168592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6690.472181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6690.541802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6690.657429] Lustre: lustre-MDT0000: new disk, initializing [ 6690.775227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6690.802598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6697.092903] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6714.795774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6715.053785] Lustre: 179718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6715.131402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6715.143551] Lustre: Skipped 1 previous similar message [ 6715.273974] Lustre: lustre-MDT0001: new disk, initializing [ 6715.374970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6715.448874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6715.467720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6722.214859] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6729.024241] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6741.716403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6742.026577] Lustre: lustre-OST0000: new disk, initializing [ 6742.031784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6742.044535] Lustre: 181656:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6742.165040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6749.925975] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6751.843448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6751.860070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6751.996275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6766.551467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6766.738403] Lustre: lustre-OST0001: new disk, initializing [ 6766.740870] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6766.769476] Lustre: 182682:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6766.935890] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6775.466379] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6776.429587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6776.448827] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6776.591894] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6790.146319] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6802.660839] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6811.088706] Lustre: DEBUG MARKER: === sanity-lfsck: finish setup 02:30:07 (1787466607) === [ 6813.418657] Lustre: DEBUG MARKER: == sanity-lfsck test complete, duration 6458 sec ========= 02:30:09 (1787466609) [ 6816.151802] Lustre: DEBUG MARKER: === sanity-lfsck: start cleanup 02:30:11 (1787466611) === [ 6820.506658] Lustre: DEBUG MARKER: === sanity-lfsck: finish cleanup 02:30:16 (1787466616) === [ 6828.001033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6828.013317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6832.130767] Lustre: server umount lustre-MDT0000 complete [ 6838.241539] LustreError: 179727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6838.258475] LustreError: 179727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6842.543296] LustreError: 179707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787466639 with bad export cookie 1633974992328976758 [ 6842.543861] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6842.559882] LustreError: 179707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6843.042732] Lustre: server umount lustre-MDT0001 complete [ 6859.679203] Lustre: 176870:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787466640/real 1787466640] req@ffff98bbb3b18a80 x1874294358498688/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787466656 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6859.735792] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6859.768421] Lustre: Skipped 3 previous similar messages [ 6863.839522] Lustre: 176873:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787466644/real 1787466644] req@ffff98ba94487b80 x1874294358498944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787466660 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6864.941084] Lustre: 176870:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787466645/real 1787466645] req@ffff98bbb764c380 x1874294358499200/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787466661 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6865.146817] Lustre: server umount lustre-OST0000 complete [ 6868.006359] Lustre: 176873:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787466649/real 1787466649] req@ffff98ba94484000 x1874294358499584/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787466665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6873.185246] Lustre: 176873:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787466654/real 1787466654] req@ffff98ba94485880 x1874294358499840/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787466670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6875.228153] Lustre: server umount lustre-OST0001 complete [ 6902.264551] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6907.055266] Key type lgssc unregistered [ 6907.653312] LNet: 186164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6907.660361] LNetError: 186164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6907.682975] LNet: Removed LNI 192.168.201.118@tcp [ 6908.891271] Key type .llcrypt unregistered [ 6908.895570] Key type ._llcrypt unregistered