[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 500112755 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2560MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002374] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008009] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.011042] Yama: becoming mindful. [ 0.012029] SELinux: Initializing. [ 0.013067] *** VALIDATE selinux *** [ 0.021399] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026726] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030044] *** VALIDATE tmpfs *** [ 0.031407] *** VALIDATE proc *** [ 0.033050] *** VALIDATE cgroup *** [ 0.034007] *** VALIDATE cgroup2 *** [ 0.035232] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042409] debug: unmapping init [mem 0xffffffff9f259000-0xffffffff9f260fff] [ 0.045160] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046902] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048009] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050009] ... value mask: 0000ffffffffffff [ 0.051011] ... max period: 00007fffffffffff [ 0.052010] ... fixed-purpose events: 3 [ 0.053008] ... event mask: 000000070000000f [ 0.055317] rcu: Hierarchical SRCU implementation. [ 0.057482] smp: Bringing up secondary CPUs ... [ 0.058586] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.062591] smp: Brought up 1 node, 4 CPUs [ 0.064010] smpboot: Max logical packages: 1 [ 0.065017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.125533] node 0 deferred pages initialised in 57ms [ 0.128137] devtmpfs: initialized [ 0.129336] x86/mm: Memory block size: 128MB [ 0.132924] gcov: version magic: 0x41383552 [ 0.135276] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.138097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141371] pinctrl core: initialized pinctrl subsystem [ 0.143136] [ 0.143641] ************************************************************* [ 0.145009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148017] ** ** [ 0.150008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153019] ** ** [ 0.155010] ** This means that this kernel is built to expose internal ** [ 0.158011] ** IOMMU data structures, which may compromise security on ** [ 0.160016] ** your system. ** [ 0.161010] ** ** [ 0.163008] ** If you see this message and you are not debugging the ** [ 0.165009] ** kernel, report this immediately to your vendor! ** [ 0.167009] ** ** [ 0.169009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171007] ************************************************************* [ 0.174235] NET: Registered protocol family 16 [ 0.176495] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.182050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186031] cpuidle: using governor menu [ 0.187572] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.189547] PCI: Using configuration type 1 for base access [ 0.191109] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.201107] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.203039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.207078] cryptd: max_cpu_qlen set to 1000 [ 0.211211] ACPI: Added _OSI(Module Device) [ 0.212008] ACPI: Added _OSI(Processor Device) [ 0.214008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.215008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.220735] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226525] ACPI: Interpreter enabled [ 0.228044] ACPI: PM: (supports S0 S3 S4 S5) [ 0.229008] ACPI: Using IOAPIC for interrupt routing [ 0.231088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.233352] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245573] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.248026] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253124] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.257560] acpiphp: Slot [2] registered [ 0.259117] acpiphp: Slot [5] registered [ 0.261119] acpiphp: Slot [6] registered [ 0.262115] acpiphp: Slot [7] registered [ 0.264089] acpiphp: Slot [8] registered [ 0.265073] acpiphp: Slot [9] registered [ 0.266126] acpiphp: Slot [10] registered [ 0.268057] acpiphp: Slot [3] registered [ 0.269075] acpiphp: Slot [4] registered [ 0.271146] acpiphp: Slot [11] registered [ 0.274086] acpiphp: Slot [12] registered [ 0.275069] acpiphp: Slot [13] registered [ 0.277111] acpiphp: Slot [14] registered [ 0.279098] acpiphp: Slot [15] registered [ 0.280079] acpiphp: Slot [16] registered [ 0.282077] acpiphp: Slot [17] registered [ 0.283075] acpiphp: Slot [18] registered [ 0.285142] acpiphp: Slot [19] registered [ 0.286086] acpiphp: Slot [20] registered [ 0.288128] acpiphp: Slot [21] registered [ 0.289083] acpiphp: Slot [22] registered [ 0.291082] acpiphp: Slot [23] registered [ 0.293083] acpiphp: Slot [24] registered [ 0.294075] acpiphp: Slot [25] registered [ 0.296141] acpiphp: Slot [26] registered [ 0.297131] acpiphp: Slot [27] registered [ 0.298104] acpiphp: Slot [28] registered [ 0.300072] acpiphp: Slot [29] registered [ 0.301062] acpiphp: Slot [30] registered [ 0.302125] acpiphp: Slot [31] registered [ 0.304069] PCI host bridge to bus 0000:00 [ 0.305012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.307014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.309015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.314016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.316015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.318151] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.320961] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.323208] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.333777] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.338057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.340012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.344012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.346463] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.349881] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.353040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.355853] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.376022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.382013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.388899] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.397015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.407013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.434015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.444000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.455000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.465013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.495016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.506074] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.514013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.527014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.553014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.571190] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.580013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.589013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.618015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.633861] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.650014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.664016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.691014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.706684] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.718014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.729012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.763014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.781009] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.784420] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.790390] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.793330] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.795194] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.807032] iommu: Default domain type: Passthrough [ 0.809473] SCSI subsystem initialized [ 0.810100] ACPI: bus type USB registered [ 0.811103] usbcore: registered new interface driver usbfs [ 0.813064] usbcore: registered new interface driver hub [ 0.815069] usbcore: registered new device driver usb [ 0.817154] pps_core: LinuxPPS API ver. 1 registered [ 0.819009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.821075] PTP clock support registered [ 0.823110] EDAC MC: Ver: 3.0.0 [ 0.825551] PCI: Using ACPI for IRQ routing [ 0.827910] NetLabel: Initializing [ 0.830012] NetLabel: domain hash size = 128 [ 0.831010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.833166] NetLabel: unlabeled traffic allowed by default [ 0.836133] vgaarb: loaded [ 0.838244] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.841012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.848004] clocksource: Switched to clocksource kvm-clock [ 1.114608] VFS: Disk quotas dquot_6.6.0 [ 1.116504] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.118789] *** VALIDATE ramfs *** [ 1.120040] *** VALIDATE hugetlbfs *** [ 1.121603] pnp: PnP ACPI init [ 1.125349] pnp: PnP ACPI: found 6 devices [ 1.142195] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.146072] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.149272] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.151749] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.154826] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.157586] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.160851] NET: Registered protocol family 2 [ 1.163415] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.168435] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.172236] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.178856] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.182234] TCP: Hash tables configured (established 65536 bind 65536) [ 1.185638] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.188808] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.191943] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.195571] NET: Registered protocol family 1 [ 1.198406] RPC: Registered named UNIX socket transport module. [ 1.201736] RPC: Registered udp transport module. [ 1.203513] RPC: Registered tcp transport module. [ 1.205322] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.208291] NET: Registered protocol family 44 [ 1.210287] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.213236] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.215853] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.218349] PCI: CLS 0 bytes, default 64 [ 1.220720] Unpacking initramfs... [ 2.625721] debug: unmapping init [mem 0xffffa0363cc54000-0xffffa0363ffbffff] [ 2.629742] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.631948] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.636421] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.156844] Initialise system trusted keyrings [ 3.158668] Key type blacklist registered [ 3.160455] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.168914] zbud: loaded [ 3.172438] *** VALIDATE nfs *** [ 3.173464] *** VALIDATE nfs4 *** [ 3.175102] pstore: using deflate compression [ 3.178632] Platform Keyring initialized [ 3.285265] NET: Registered protocol family 38 [ 3.286423] Key type asymmetric registered [ 3.287483] Asymmetric key parser 'x509' registered [ 3.290429] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.297278] io scheduler mq-deadline registered [ 3.299245] io scheduler kyber registered [ 3.301719] io scheduler bfq registered [ 3.303669] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.305986] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.308708] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.312654] ACPI: Power Button [PWRF] [ 3.319445] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.325389] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.343357] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.353884] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.379436] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.408212] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.437605] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.443873] Non-volatile memory driver v1.3 [ 3.447431] Linux agpgart interface v0.103 [ 3.481551] virtio_blk virtio1: [vda] 147232 512-byte logical blocks (75.4 MB/71.9 MiB) [ 3.484786] vda: detected capacity change from 0 to 75382784 [ 3.502477] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.506237] vdb: detected capacity change from 0 to 1073741824 [ 3.520740] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.524216] vdc: detected capacity change from 0 to 2621440000 [ 3.545493] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.548645] vdd: detected capacity change from 0 to 2621440000 [ 3.570216] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.573691] vde: detected capacity change from 0 to 4294967296 [ 3.592333] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.595765] vdf: detected capacity change from 0 to 4294967296 [ 3.602673] libphy: Fixed MDIO Bus: probed [ 3.607837] usbcore: registered new interface driver usbserial_generic [ 3.610628] usbserial: USB Serial support registered for generic [ 3.613458] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.618542] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.630716] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.636777] mousedev: PS/2 mouse device common for all mice [ 3.640960] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.642354] rtc_cmos 00:05: RTC can wake from S4 [ 3.653038] rtc_cmos 00:05: registered as rtc0 [ 3.654621] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.655524] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.661611] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.662802] intel_pstate: CPU model not supported [ 3.669526] hid: raw HID events driver (C) Jiri Kosina [ 3.671895] usbcore: registered new interface driver usbhid [ 3.674204] usbhid: USB HID core driver [ 3.675764] drop_monitor: Initializing network drop monitor service [ 3.678397] Initializing XFRM netlink socket [ 3.680504] NET: Registered protocol family 10 [ 3.683730] Segment Routing with IPv6 [ 3.685394] NET: Registered protocol family 17 [ 3.687553] mpls_gso: MPLS GSO support [ 3.693642] RAS: Correctable Errors collector initialized. [ 3.696163] AVX version of gcm_enc/dec engaged. [ 3.697960] AES CTR mode by8 optimization enabled [ 3.784541] sched_clock: Marking stable (3784508776, 0)->(4727921944, -943413168) [ 3.787963] registered taskstats version 1 [ 3.789561] Loading compiled-in X.509 certificates [ 3.791911] zswap: loaded using pool lzo/zbud [ 3.818352] Key type big_key registered [ 3.831860] Key type encrypted registered [ 3.834419] ima: No TPM chip found, activating TPM-bypass! [ 3.836921] ima: Allocated hash algorithm: sha1 [ 3.838694] ima: No architecture policies found [ 3.841223] evm: Initialising EVM extended attributes: [ 3.843217] evm: security.selinux [ 3.844481] evm: security.ima [ 3.845660] evm: security.capability [ 3.847132] evm: HMAC attrs: 0x1 [ 3.849741] rtc_cmos 00:05: setting system clock to 2026-09-11 10:32:59 UTC (1789122779) [ 3.856539] debug: unmapping init [mem 0xffffffffa0203000-0xffffffffa03fffff] [ 3.859896] debug: unmapping init [mem 0xffffffff9ef82000-0xffffffff9f258fff] [ 3.869210] Write protecting the kernel read-only data: 28672k [ 3.872864] debug: unmapping init [mem 0xffffffff9d603000-0xffffffff9d7fffff] [ 3.876043] debug: unmapping init [mem 0xffffffff9df14000-0xffffffff9dffffff] [ 3.921084] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.933328] systemd[1]: Detected virtualization kvm. [ 3.935670] systemd[1]: Detected architecture x86-64. [ 3.937906] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.983354] systemd[1]: No hostname configured. [ 3.985290] systemd[1]: Set hostname to . [ 3.989264] random: systemd: uninitialized urandom read (16 bytes read) [ 3.993299] systemd[1]: Initializing machine ID from random generator. [ 4.054116] random: ln: uninitialized urandom read (6 bytes read) [ 4.170803] random: systemd: uninitialized urandom read (16 bytes read) [ 4.174446] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.180337] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.189315] systemd[1]: Starting Create Volatile Files and Directories... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook.[ 5.177469] random: fast init done Starting dracut pre-udev hook... [ 5.238664] device-mapper: uevent: version 1.0.3 [ 5.240976] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.208324] virtio_net virtio0 ens2: renamed from eth0 [ 7.541549] scsi host0: ata_piix [ 7.614335] scsi host1: ata_piix [ 7.615758] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.622331] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.352310] random: crng init done [ 12.354666] random: 7 urandom warning(s) missed due to ratelimiting [ 14.831509] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.602817] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.157732] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.615415] SELinux: Disabled at runtime. [ 21.814486] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.844160] systemd[1]: Detected virtualization kvm. [ 21.848631] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.383451] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.394771] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.419150] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.440644] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.452869] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.498228] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.555388] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ 25.097887] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.291769] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.370110] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.658376] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.035756] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.093139] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 33.567545] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 34.463436] NFS: Registering the id_resolver key type [ 34.470713] Key type id_resolver registered [ 34.475874] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg340-server login: [ 75.007693] spl: loading out-of-tree module taints kernel. [ 80.838725] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 92.499732] Key type ._llcrypt registered [ 92.507798] Key type .llcrypt registered [ 92.630204] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_hostid [ 109.158221] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [ 110.497067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 110.511421] alg: No test for adler32 (adler32-zlib) [ 111.466472] hrtimer: interrupt took 5462174 ns [ 111.990463] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 112.889627] LNet: Added LNI 192.168.203.140@tcp [8/256/0/180] [ 114.711587] Key type lgssc registered [ 116.778663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 127.077075] vdc: vdc1 vdc9 [ 135.627507] vde: vde1 vde9 [ 147.495346] vdf: vdf1 vdf9 [ 171.605085] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [ 182.652611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 184.043660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 184.315884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 184.379521] Lustre: lustre-MDT0000: new disk, initializing [ 184.715274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.782353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 188.601356] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.191210] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 199.018954] Lustre: lustre-OST0000: new disk, initializing [ 199.022446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.025936] Lustre: Skipped 1 previous similar message [ 199.130922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.250184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 205.274030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 205.520892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 207.009178] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.598425] Lustre: lustre-OST0001: new disk, initializing [ 220.608842] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 220.797537] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 229.640788] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.226680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 230.235121] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 230.464872] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 243.142066] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 250.697822] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 258.714598] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing check_logdir /tmp/testlogs/ [ 263.408905] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing yml_node [ 267.146791] Lustre: DEBUG MARKER: Client: 2.17.57.138 [ 269.791645] Lustre: DEBUG MARKER: MDS: 2.17.57.138 [ 273.240351] Lustre: DEBUG MARKER: OSS: 2.17.57.138 [ 275.564881] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Fri Sep 11 06:37:29 EDT 2026 [ 294.971844] Lustre: DEBUG MARKER: - need MDS1_VERSION <= 2.16.61-1-g89cf292a8c2 (34683274 <= 34618625) for LU-18938, skip 360 [ 297.134890] Lustre: DEBUG MARKER: - need MDS1_VERSION < v2_14_55-100-g8a84c7f9c7 (34683274 < 34486116) for LU-14927, skip 0f [ 299.099658] Lustre: DEBUG MARKER: - need OST1_VERSION < v2_17_50-410-gbd8e7b9555 (34683274 < 34681754) for LU-12550, skip 216 [ 300.796467] Lustre: DEBUG MARKER: excepting tests: 42a 42c 42b 118c 118d 407 119i 216 817 411a 130b 130c 130d 130e 130f 130g [ 302.077818] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 51b 51c 51e 834 [ 303.738938] Lustre: DEBUG MARKER: === sanity: start setup 06:37:57 (1789123077) === [ 309.223962] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing check_config_client /mnt/lustre [ 325.375507] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 328.320409] Lustre: 11522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 331.947914] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 336.363269] Lustre: Setting parameter lustre.lod.lustre-MDT0000-mdtlov.enable_compr_rotational=1 in log params [ 341.157491] Lustre: DEBUG MARKER: === sanity: finish setup 06:38:35 (1789123115) === [ 347.216769] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 06:38:41 (1789123121) [ 350.692211] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 352.101202] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 06:38:46 (1789123126) [ 358.805441] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 06:38:53 (1789123133) [ 527.367174] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 06:41:41 (1789123301) [ 533.516251] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 06:41:48 (1789123308) [ 534.847253] Lustre: *** cfs_fail_loc=15b, val=0*** [ 534.850419] Lustre: *** cfs_fail_loc=15b, val=0*** [ 534.852859] LustreError: 7183:0:(llog_cat_server.c:483:llog_cat_add_rec()) lustre-OST0001-osc-MDT0000: initialization error: rc = -28 [ 541.023550] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 06:41:55 (1789123315) [ 548.951891] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 06:42:03 (1789123323) [ 550.284912] Lustre: *** cfs_fail_loc=19a, val=0*** [ 551.180725] Lustre: *** cfs_fail_loc=19a, val=0*** [ 553.065922] Lustre: *** cfs_fail_loc=19a, val=0*** [ 553.067757] Lustre: Skipped 1 previous similar message [ 555.211839] Lustre: *** cfs_fail_loc=19a, val=0*** [ 555.215487] Lustre: Skipped 1 previous similar message [ 559.966724] Lustre: *** cfs_fail_loc=19a, val=0*** [ 559.970628] Lustre: Skipped 4 previous similar messages [ 568.388656] Lustre: *** cfs_fail_loc=19a, val=0*** [ 568.394871] Lustre: Skipped 8 previous similar messages [ 584.867259] Lustre: *** cfs_fail_loc=19a, val=0*** [ 584.871485] Lustre: Skipped 18 previous similar messages [ 617.301292] Lustre: *** cfs_fail_loc=19a, val=0*** [ 617.310959] Lustre: Skipped 36 previous similar messages [ 652.612257] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 06:43:46 (1789123426) [ 654.359937] Lustre: DEBUG MARKER: SKIP: sanity test_60h Need at least 2 MDTs [ 655.846449] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 657.456964] Lustre: DEBUG MARKER: == sanity test 60j: llog_reader reports corruptions ====== 06:43:51 (1789123431) [ 659.104996] Lustre: DEBUG MARKER: SKIP: sanity test_60j ldiskfs only test [ 661.115034] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 06:43:55 (1789123435) [ 667.010185] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 06:44:01 (1789123441) [ 673.156228] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 06:44:07 (1789123447) [ 741.616414] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 06:45:15 (1789123515) [ 754.164488] Lustre: DEBUG MARKER: == sanity test 63c: test sync_on_close=1 ================= 06:45:28 (1789123528) [ 773.105314] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 06:45:47 (1789123547) [ 782.191604] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 784.192924] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 06:45:58 (1789123558) [ 793.168163] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 06:46:07 (1789123567) [ 855.997281] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 06:47:10 (1789123630) [ 861.453146] Lustre: *** cfs_fail_loc=725, val=0*** [ 867.762288] Lustre: *** cfs_fail_loc=725, val=0*** [ 877.354903] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 06:47:31 (1789123651) [ 891.657402] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 06:47:45 (1789123665) [ 943.364578] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 06:48:37 (1789123717) [ 963.004270] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 06:48:57 (1789123737) [ 971.700349] Lustre: *** cfs_fail_loc=513, val=17*** [ 971.702273] LustreError: 13097:0:(service.c:2341:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1876031311163392 [ 973.649927] Lustre: Failing over lustre-OST0000 [ 975.828215] Lustre: server umount lustre-OST0000 complete [ 977.888214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.440177] LustreError: 13081:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 981.467672] LustreError: 13081:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 983.012069] LustreError: 13080:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 986.563030] LustreError: 6711:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 991.683595] LustreError: 13077:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 991.706905] LustreError: 13077:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 993.485168] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 993.503798] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 994.763683] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 995.250601] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 995.257498] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 999.231309] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [ 1007.975108] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1009.593741] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 1020.474292] Lustre: DEBUG MARKER: == sanity test 64j: check grants on re-done rpc ========== 06:49:54 (1789123794) [ 1021.441425] Lustre: *** cfs_fail_loc=256, val=0*** [ 1028.828847] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 06:50:03 (1789123803) [ 1033.941220] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 06:50:08 (1789123808) [ 1039.592794] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 06:50:14 (1789123814) [ 1045.093982] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 06:50:19 (1789123819) [ 1050.534991] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 06:50:24 (1789123824) [ 1057.785306] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 06:50:31 (1789123831) [ 1063.897466] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 06:50:38 (1789123838) [ 1071.042924] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 06:50:45 (1789123845) [ 1077.111507] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 06:50:51 (1789123851) [ 1084.731429] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 06:50:58 (1789123858) [ 1091.419709] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 06:51:06 (1789123866) [ 1092.896473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.909166] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1092.916802] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1093.763239] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1109.841268] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1122.695989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.706526] Lustre: Skipped 1 previous similar message [ 1122.712170] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1122.721376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1122.733661] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1122.749136] Lustre: Skipped 1 previous similar message [ 1127.582395] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1127.735724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1146.576692] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1161.149770] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.160607] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1161.166476] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1161.179962] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1165.311452] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1165.511962] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1171.418344] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 06:52:25 (1789123945) [ 1177.918881] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 06:52:31 (1789123951) [ 1184.517707] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 06:52:39 (1789123959) [ 1195.029105] Lustre: DEBUG MARKER: SKIP: sanity test_65n needs >= 2 MDTs [ 1208.705625] Lustre: DEBUG MARKER: == sanity test 65o: pool inheritance for mdt component === 06:53:03 (1789123983) [ 1232.118494] Lustre: DEBUG MARKER: == sanity test 65p: setstripe with yaml file and huge number ========================================================== 06:53:26 (1789124006) [ 1238.365949] Lustre: DEBUG MARKER: == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 06:53:32 (1789124012) [ 1244.092557] Lustre: DEBUG MARKER: == sanity test 65r: prevent all-zero offsets ============= 06:53:38 (1789124018) [ 1250.147248] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client === 06:53:44 (1789124024) [ 1262.536305] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 06:53:56 (1789124036) [ 1264.286156] Lustre: *** cfs_fail_loc=217, val=0*** [ 1264.293219] Lustre: Skipped 2 previous similar messages [ 1266.888206] Lustre: *** cfs_fail_loc=217, val=0*** [ 1266.893964] Lustre: Skipped 4 previous similar messages [ 1273.457262] Lustre: DEBUG MARKER: == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 06:54:08 (1789124048) [ 1287.939129] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 1289.869625] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 06:54:24 (1789124064) [ 1299.200512] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 06:54:33 (1789124073) [ 1309.804150] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 06:54:43 (1789124083) [ 1342.571629] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 06:55:17 (1789124117) [ 1348.029686] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 06:55:22 (1789124122) [ 1353.312952] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 06:55:27 (1789124127) [ 1359.454875] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 06:55:33 (1789124133) [ 1432.095113] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 06:56:46 (1789124206) [ 1479.037881] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 06:57:33 (1789124253) [ 1485.515114] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 06:57:39 (1789124259) [ 1485.912047] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.40@tcp inode [0x200000406:0xc34:0x0] object 0x280000400:3788 extent [0-1048575]: client csum 45e17b06, server csum 45e17b05 [ 1488.424122] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1490.059249] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.40@tcp inode [0x200000406:0xc34:0x0] object 0x280000400:3788 extent [1048576-2097151], client returned csum e2348287 (type 1), server csum 8b2308f9 (type 1) [ 1492.935015] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1494.782106] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.40@tcp inode [0x200000406:0xc34:0x0] object 0x280000400:3788 extent [0-1048575], client returned csum 212dcb1a (type 2), server csum bc82cbc1 (type 2) [ 1497.598664] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1498.572490] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.40@tcp inode [0x200000406:0xc34:0x0] object 0x280000400:3788 extent [0-1048575], client returned csum 2ed66539 (type 4), server csum 45e17b05 (type 4) [ 1501.237591] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1502.785312] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.40@tcp inode [0x200000406:0xc34:0x0] object 0x280000400:3788 extent [1048576-2097151], client returned csum fbc9f878 (type 10), server csum cbe7f7d5 (type 10) [ 1505.384445] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1506.818289] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.40@tcp inode [0x200000406:0xc34:0x0] object 0x280000400:3788 extent [0-1048575], client returned csum 76070442 (type 20), server csum 281b039b (type 20) [ 1509.348692] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1512.942626] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1516.560524] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1523.210556] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 06:58:17 (1789124297) [ 1532.353780] Lustre: 33071:0:(tgt_handler.c:2013:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000406:0xc35:0x0]:[0-1048575]-2ed66539-45e17b05 [ 1532.375290] LustreError: dumping log to /tmp/lustre-log.1789124308.33071 [ 1532.525792] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.40@tcp inode [0x200000406:0xc35:0x0] object 0x240000400:3798 extent [0-1048575], client returned csum 2ed66539 (type 4), server csum 45e17b05 (type 4) [ 1532.543333] LustreError: Skipped 2 previous similar messages [ 1564.848420] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 06:58:59 (1789124339) [ 1565.211818] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.40@tcp inode [0x200000406:0xc37:0x0] object 0x240000400:3799 extent [0-1048575]: client csum b5ea7f3c, server csum b5ea7f3b [ 1568.192675] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.40@tcp inode [0x200000406:0xc37:0x0] object 0x240000400:3799 extent [0-1048575], client returned csum fe0f685f (type 4), server csum b5ea7f3b (type 4) [ 1575.823858] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 06:59:09 (1789124349) [ 1577.478411] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1577.819962] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.40@tcp inode [0x200000406:0xc38:0x0] object 0x280000400:3789 extent [0-1048575]: client csum b2f1b12, server csum b2f1b11 [ 1581.406755] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1581.894115] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.40@tcp inode [0x200000406:0xc39:0x0] object 0x240000400:3800 extent [0-1048575]: client csum 19eeae62, server csum 19eeae61 [ 1581.925097] LustreError: Skipped 16 previous similar messages [ 1585.275539] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1586.867642] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.40@tcp inode [0x200000406:0xc3a:0x0] object 0x280000400:3790 extent [3145728-4194303]: client csum b5ea7f3c, server csum b5ea7f3b [ 1586.895088] LustreError: Skipped 22 previous similar messages [ 1588.994474] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1592.746809] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1596.384634] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1596.779437] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.40@tcp inode [0x200000406:0xc3d:0x0] object 0x240000400:3802 extent [0-1048575]: client csum 887488b6, server csum 887488b5 [ 1596.805822] LustreError: Skipped 39 previous similar messages [ 1599.647443] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1607.844406] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1609.946775] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 06:59:43 (1789124383) [ 1611.647636] Lustre: *** cfs_fail_loc=21a, val=0*** [ 1617.367077] Lustre: *** cfs_fail_loc=21b, val=0*** [ 1631.608212] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 07:00:05 (1789124405) [ 1632.912260] Lustre: Setting parameter lustre.osc.lustre*.checksums=0 in log params [ 1638.943666] Lustre: Modifying parameter lustre.osc.lustre*.checksums=1 in log params [ 1644.777785] Lustre: Disabling parameter lustre.osc.lustre*.checksums= in log params [ 1658.409716] Lustre: Setting parameter lustre.osc.lustre*.checksums=0 in log params [ 1662.385256] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 07:00:36 (1789124436) [ 1664.134178] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 1665.711514] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1671.197311] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid 50 [ 1672.494685] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid in IDLE state after 0 sec [ 1677.309537] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid 50 [ 1678.858576] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid in FULL state after 0 sec [ 1680.522347] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1687.317503] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid 50 [ 1695.453669] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid in IDLE state after 6 sec [ 1700.675961] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid 50 [ 1702.133848] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid in FULL state after 0 sec [ 1703.893590] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1709.556767] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid 50 [ 1716.656747] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid in IDLE state after 5 sec [ 1722.212948] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid 50 [ 1723.540662] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid in FULL state after 0 sec [ 1725.118907] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1730.536228] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid 50 [ 1741.970184] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid in IDLE state after 9 sec [ 1746.956359] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid 50 [ 1748.360506] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid in FULL state after 0 sec [ 1749.777939] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1755.668481] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid 50 [ 1762.403455] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid in IDLE state after 5 sec [ 1767.090574] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid 50 [ 1768.310379] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid in FULL state after 0 sec [ 1769.752022] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1775.688566] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid 50 [ 1782.333299] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid in IDLE state after 5 sec [ 1787.273850] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid 50 [ 1788.724822] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid in FULL state after 0 sec [ 1790.553249] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1795.840296] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid 50 [ 1802.527772] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid in IDLE state after 5 sec [ 1807.622918] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid 50 [ 1809.267905] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac487e80000.ost_server_uuid in FULL state after 0 sec [ 1815.959646] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1817.665297] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 07:03:11 (1789124591) [ 1822.706414] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 07:03:17 (1789124597) [ 1824.441353] Lustre: DEBUG MARKER: SKIP: sanity test_77n f77n.sanity blocks not contiguous around hole [ 1826.180987] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 07:03:20 (1789124600) [ 1837.863449] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 07:03:31 (1789124611) [ 1848.109894] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check =========== 07:03:42 (1789124622) [ 1875.462823] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 07:04:09 (1789124649) [ 1884.501480] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 07:04:18 (1789124658) [ 1885.971597] Lustre: *** cfs_fail_loc=228, val=0*** [ 1892.286885] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 07:04:26 (1789124666) [ 1893.669855] Lustre: *** cfs_fail_loc=228, val=0*** [ 1900.897855] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 07:04:34 (1789124674) [ 1929.419690] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ========================================================== 07:05:03 (1789124703) [ 1938.173454] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 07:05:12 (1789124712) [ 2097.956174] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 07:07:52 (1789124872) [ 2126.966461] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 07:08:21 (1789124901) [ 2197.465627] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 07:09:31 (1789124971) [ 2475.492571] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 07:14:09 (1789125249) [ 2561.229172] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 07:15:35 (1789125335) [ 2573.365232] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 07:15:47 (1789125347) [ 2666.628833] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 07:17:20 (1789125440) [ 2687.823560] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 07:17:42 (1789125462) [ 2698.162673] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 07:17:52 (1789125472) [ 2753.242940] Lustre: DEBUG MARKER: == sanity test 101m: read ahead for small file and last stripe of the file ========================================================== 07:18:47 (1789125527) [ 2754.878734] Lustre: DEBUG MARKER: SKIP: sanity test_101m need >= 2.13.57 and ldiskfs for fallocate [ 2756.469529] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 07:18:50 (1789125530) [ 2763.636921] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 07:18:57 (1789125537) [ 2775.181869] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 07:19:09 (1789125549) [ 2782.721521] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 07:19:17 (1789125557) [ 2800.825343] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 07:19:34 (1789125574) [ 2819.923418] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 07:19:54 (1789125594) [ 2822.436500] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 2824.209247] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 2825.818629] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 2827.567362] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2833.736830] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 07:20:08 (1789125608) [ 2835.889718] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2837.371724] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 2838.924238] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 2840.525649] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2842.924152] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2849.229258] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 07:20:23 (1789125623) [ 2855.866313] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 07:20:30 (1789125630) [ 2875.535763] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 07:20:49 (1789125649) [ 2884.906179] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 07:20:58 (1789125658) [ 2894.029341] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 07:21:08 (1789125668) [ 2901.797821] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 07:21:15 (1789125675) [ 2911.246128] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 07:21:25 (1789125685) [ 2919.225748] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 07:21:32 (1789125692) [ 2926.077504] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 07:21:40 (1789125700) [ 2933.565722] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 07:21:47 (1789125707) [ 2941.956692] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 07:21:56 (1789125716) [ 2951.124140] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 07:22:04 (1789125724) [ 3232.112400] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 07:26:46 (1789126006) [ 3400.184344] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 07:29:33 (1789126173) [ 3407.559953] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 07:29:41 (1789126181) [ 3434.560536] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 07:30:08 (1789126208) [ 3437.725588] Lustre: lustre-MDD0000: changelog on [ 3445.004592] Lustre: lustre-MDD0000: changelog off [ 3447.603951] Lustre: DEBUG MARKER: == sanity test 103g: no MDS_GETXATTR storm for inodes without ACL_ACCESS (LU-17238) ========================================================== 07:30:22 (1789126222) [ 3454.854402] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 07:30:29 (1789126229) [ 3455.679972] Lustre: lustre-OST0000: Client 36b72586-0d6d-4842-91e5-74b6ec53e0b3 (at 192.168.203.40@tcp) reconnecting [ 3461.290850] Lustre: DEBUG MARKER: oleg340-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9ac3b32d1800.ost_server_uuid 50 [ 3462.728420] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac3b32d1800.ost_server_uuid in FULL state after 0 sec [ 3469.135924] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 07:30:43 (1789126243) [ 3475.249604] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 07:30:49 (1789126249) [ 3496.361779] Lustre: DEBUG MARKER: == sanity test 104d: runas -u 500 -g 500 lctl dl test ==== 07:31:10 (1789126270) [ 3502.650511] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 07:31:16 (1789126276) [ 3508.753198] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 07:31:23 (1789126283) [ 3515.241410] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 07:31:29 (1789126289) [ 3521.776589] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 07:31:35 (1789126295) [ 3541.518961] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 07:31:55 (1789126315) [ 3549.823468] Lustre: DEBUG MARKER: == sanity test 105f: Enqueue same range flocks =========== 07:32:03 (1789126323) [ 3560.137696] Lustre: DEBUG MARKER: == sanity test 105g: ldlm_lock_debug stack test ========== 07:32:14 (1789126334) [ 3570.478535] Lustre: DEBUG MARKER: == sanity test 105h: Flock functional verify ============= 07:32:24 (1789126344) [ 3577.627429] Lustre: DEBUG MARKER: == sanity test 105i: Flock deadlock verify =============== 07:32:32 (1789126352) [ 3589.645386] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 07:32:43 (1789126363) [ 3598.391923] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 07:32:52 (1789126372) [ 3609.012174] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 07:33:03 (1789126383) [ 3616.024781] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 07:33:10 (1789126390) [ 3856.411108] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 07:37:10 (1789126630) [ 3858.917115] Lustre: *** cfs_fail_loc=147, val=0*** [ 3867.050347] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 07:37:21 (1789126641) [ 3873.015655] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 07:37:27 (1789126647) [ 3879.276606] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 07:37:33 (1789126653) [ 3881.076790] Lustre: *** cfs_fail_loc=217, val=0*** [ 3881.093745] Lustre: *** cfs_fail_loc=217, val=0*** [ 3881.095641] Lustre: Skipped 1 previous similar message [ 3887.381685] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 3888.594409] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 3889.843466] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 07:37:44 (1789126664) [ 3895.952847] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 07:37:50 (1789126670) [ 3902.239319] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 07:37:56 (1789126676) [ 3904.005081] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3905.087499] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3907.135095] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3910.208822] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3914.242548] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3923.712425] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 07:38:17 (1789126697) [ 3926.554755] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3940.083459] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 07:38:34 (1789126714) [ 3948.360759] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 07:38:42 (1789126722) [ 3950.261866] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3950.267650] Lustre: Skipped 3 previous similar messages [ 3969.248701] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 07:39:03 (1789126743) [ 3979.924948] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 07:39:13 (1789126753) [ 3988.412288] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 07:39:22 (1789126762) [ 3992.545271] LustreError: 29330:0:(ofd_dev.c:1931:ofd_statfs_hdl()) cfs_fail_timeout id 242 sleeping for 10000ms [ 3992.967655] LustreError: 29330:0:(ofd_dev.c:1931:ofd_statfs_hdl()) cfs_fail_timeout interrupted [ 3998.591187] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 07:39:33 (1789126773) [ 4004.354849] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 07:39:38 (1789126778) [ 4010.564352] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 07:39:44 (1789126784) [ 4016.757527] Lustre: DEBUG MARKER: == sanity test 119e: Basic tests of dio read and write at various sizes ========================================================== 07:39:51 (1789126791) [ 4051.751487] Lustre: DEBUG MARKER: == sanity test 119f: dio vs dio race ===================== 07:40:26 (1789126826) [ 4093.802923] Lustre: DEBUG MARKER: == sanity test 119g: dio vs buffered I/O race ============ 07:41:08 (1789126868) [ 4130.518520] Lustre: DEBUG MARKER: == sanity test 119h: basic tests of memory unaligned dio ========================================================== 07:41:44 (1789126904) [ 4155.275187] Lustre: DEBUG MARKER: SKIP: sanity test_119i skipping ALWAYS excluded test 119i [ 4156.961505] Lustre: DEBUG MARKER: == sanity test 119j: basic tests of hybrid IO switching == 07:42:11 (1789126931) [ 4162.880663] Lustre: DEBUG MARKER: == sanity test 119k: hybrid IO counting with stats and disabling ========================================================== 07:42:17 (1789126937) [ 4176.611270] Lustre: DEBUG MARKER: == sanity test 119m: Test DIO readv/writev: exercise iter duplication ========================================================== 07:42:31 (1789126951) [ 4182.357850] Lustre: DEBUG MARKER: == sanity test 119n: Test Unaligned DIO readv() and writev() with unpatched ZFS ========================================================== 07:42:36 (1789126956) [ 4183.711907] Lustre: DEBUG MARKER: SKIP: sanity test_119n zfs server without 'unaligned_dio' support [ 4185.488538] Lustre: DEBUG MARKER: == sanity test 119o: Test Unaligned DIO readv() and writev() with unpatched servers ========================================================== 07:42:39 (1789126959) [ 4186.973058] Lustre: DEBUG MARKER: SKIP: sanity test_119o need ldiskfs without 'unaligned_dio' support [ 4188.519904] Lustre: DEBUG MARKER: == sanity test 119p: Test Unaligned DIO readv() and writev() with patched servers ========================================================== 07:42:43 (1789126963) [ 4194.904428] Lustre: DEBUG MARKER: == sanity test 119q: Test patchded Unaligned DIO readv() and writev() ========================================================== 07:42:49 (1789126969) [ 4208.360515] Lustre: DEBUG MARKER: == sanity test 119r: Test error handling in unaligned DIO user copy ========================================================== 07:43:02 (1789126982) [ 4214.218774] Lustre: DEBUG MARKER: == sanity test 119s: full-size unaligned DIO packs matching bulk MDs ========================================================== 07:43:08 (1789126988) [ 4215.686353] Lustre: DEBUG MARKER: SKIP: sanity test_119s need client page size larger than the server's [ 4217.098239] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 07:43:11 (1789126991) [ 4225.926130] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 07:43:20 (1789127000) [ 4233.740546] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 07:43:28 (1789127008) [ 4243.489289] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 07:43:37 (1789127017) [ 4252.248633] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 07:43:46 (1789127026) [ 4269.634845] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 07:44:04 (1789127044) [ 4287.683467] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 07:44:22 (1789127062) [ 4712.145144] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 07:51:26 (1789127486) [ 4718.310438] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 07:51:33 (1789127493) [ 4726.054262] Lustre: DEBUG MARKER: 'ls -l' 100 files without statahead: 2 sec [ 4728.269237] Lustre: DEBUG MARKER: 'ls -l' 100 files with statahead: 0 sec [ 4769.425325] Lustre: DEBUG MARKER: 'ls -l' 1000 files without statahead: 23 sec [ 4780.200878] Lustre: DEBUG MARKER: 'ls -l' 1000 files with statahead: 7 sec [ 4782.213786] Lustre: DEBUG MARKER: 'ls -l' done [ 4804.366895] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123aa.sanity: 19 seconds [ 4814.755704] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 07:53:09 (1789127589) [ 4823.547724] Lustre: DEBUG MARKER: 'statx -l' 100 files without statahead: 2 sec [ 4825.986975] Lustre: DEBUG MARKER: 'statx -l' 100 files with statahead: 1 sec [ 4865.251642] Lustre: DEBUG MARKER: 'statx -l' 1000 files without statahead: 20 sec [ 4874.179969] Lustre: DEBUG MARKER: 'statx -l' 1000 files with statahead: 6 sec [ 4875.630449] Lustre: DEBUG MARKER: 'statx -l' done [ 4896.707047] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ab.sanity: 18 seconds [ 4908.099915] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 07:54:42 (1789127682) [ 4916.036349] Lustre: DEBUG MARKER: 'statx -c 1 [ 4918.374927] Lustre: DEBUG MARKER: 'statx -c 1 [ 4947.961939] Lustre: DEBUG MARKER: 'statx -c 1 [ 4954.641489] Lustre: DEBUG MARKER: 'statx -c 1 [ 4956.591903] Lustre: DEBUG MARKER: 'statx -c 1 [ 4975.955559] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ac.sanity: 17 seconds [ 4982.512406] Lustre: DEBUG MARKER: 'statx --cached=always -D' 100 files without statahead: 0 sec [ 4984.374996] Lustre: DEBUG MARKER: 'statx --cached=always -D' 100 files with statahead: 0 sec [ 5001.729925] Lustre: DEBUG MARKER: 'statx --cached=always -D' 1000 files without statahead: 0 sec [ 5003.731709] Lustre: DEBUG MARKER: 'statx --cached=always -D' 1000 files with statahead: 0 sec [ 5125.234582] Lustre: DEBUG MARKER: 'statx --cached=always -D' 10000 files without statahead: 0 sec [ 5127.229289] Lustre: DEBUG MARKER: 'statx --cached=always -D' 10000 files with statahead: 0 sec [ 5128.527377] Lustre: DEBUG MARKER: 'statx --cached=always -D' done [ 5411.741380] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ac.sanity: 281 seconds [ 5426.280640] Lustre: DEBUG MARKER: == sanity test 123ad: Verify batching statahead works correctly ========================================================== 08:03:20 (1789128200) [ 5435.782876] Lustre: DEBUG MARKER: 'ls -l' 100 files without statahead: 2 sec [ 5438.505077] Lustre: DEBUG MARKER: 'ls -l' 100 files with statahead: 1 sec [ 5477.658777] Lustre: DEBUG MARKER: 'ls -l' 1000 files without statahead: 20 sec [ 5489.873738] Lustre: DEBUG MARKER: 'ls -l' 1000 files with statahead: 9 sec [ 5491.621589] Lustre: DEBUG MARKER: 'ls -l' done [ 5516.453469] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds [ 6118.464388] Lustre: DEBUG MARKER: 'ls -l' 100 files without statahead: 2 sec [ 6121.487722] Lustre: DEBUG MARKER: 'ls -l' 100 files with statahead: 1 sec [ 6163.201868] Lustre: DEBUG MARKER: 'ls -l' 1000 files without statahead: 22 sec [ 6173.214956] Lustre: DEBUG MARKER: 'ls -l' 1000 files with statahead: 7 sec [ 6174.838793] Lustre: DEBUG MARKER: 'ls -l' done [ 6198.169933] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds [ 6667.202955] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 08:24:01 (1789129441) [ 6695.409231] Lustre: DEBUG MARKER: ls done [ 6720.726471] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 08:24:55 (1789129495) [ 6722.033465] Lustre: DEBUG MARKER: SKIP: sanity test_123c needs >= 2 MDTs [ 6723.677652] Lustre: DEBUG MARKER: == sanity test 123d: Statahead on striped directories works correctly ========================================================== 08:24:58 (1789129498) [ 6737.095186] Lustre: DEBUG MARKER: == sanity test 123e: statahead with large wide striping == 08:25:11 (1789129511) [ 7035.495677] Lustre: DEBUG MARKER: == sanity test 123f: Retry mechanism with large wide striping files ========================================================== 08:30:07 (1789129807) [ 7157.518553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x240000400 to 0x240000401 [ 7161.399832] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x280000400 to 0x280000401 [ 7435.330519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x240000401 to 0x240000402 [ 7499.513153] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8396.548259] Lustre: DEBUG MARKER: == sanity test 123g: Test for stat-ahead advise ========== 08:52:46 (1789131166) [ 8496.326457] Lustre: DEBUG MARKER: == sanity test 123h: Verify statahead work with the fname pattern via du ========================================================== 08:54:28 (1789131268) [10042.082465] Lustre: DEBUG MARKER: == sanity test 123i: Verify statahead work with the fname indexing pattern ========================================================== 09:20:16 (1789132816) [10169.696376] Lustre: DEBUG MARKER: == sanity test 123j: -ENOENT error from batched statahead be handled correctly ========================================================== 09:22:23 (1789132943) [10171.065877] Lustre: DEBUG MARKER: SKIP: sanity test_123j needs >= 2 MDTs [10173.123885] Lustre: DEBUG MARKER: == sanity test 123k: Verify statahead work with mdtest shared stat() mode ========================================================== 09:22:27 (1789132947) [10174.932135] Lustre: DEBUG MARKER: SKIP: sanity test_123k mdtest not found [10177.023837] Lustre: DEBUG MARKER: == sanity test 123l: Avoid panic when revalidate a local cached entry ========================================================== 09:22:31 (1789132951) [10259.142020] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 09:23:53 (1789133033) [10261.319653] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [10306.294414] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff9ac4b8bd2800 [10307.787201] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff9ac4b8bd2800 [10309.217437] Lustre: DEBUG MARKER: LRU=2003 [10310.761652] Lustre: DEBUG MARKER: LIMIT=61549 [10312.300994] Lustre: DEBUG MARKER: LVF=3687400 [10314.510564] Lustre: DEBUG MARKER: OLD_LVF=100 [10316.562544] Lustre: DEBUG MARKER: Sleep 50 sec [10369.163987] Lustre: DEBUG MARKER: Dropped 1065 locks in 50s [10370.475084] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [10403.130532] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 09:26:17 (1789133177) [10507.926664] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [10700.091965] Lustre: DEBUG MARKER: ls -la time: 188 seconds [10702.318589] Lustre: DEBUG MARKER: lru_size = 400 [10923.634715] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [11040.368733] Lustre: DEBUG MARKER: ls -la time: 109 seconds [11042.265986] Lustre: DEBUG MARKER: lru_size = 8005 [11044.505260] Lustre: DEBUG MARKER: ls -la is 42% faster with lru resize enabled [11135.382612] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 09:38:28 (1789133908) [11169.307071] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize disabled ========================================================== 09:39:03 (1789133943) [11202.972642] Lustre: DEBUG MARKER: == sanity test 124e: LFRU keep priv locks from eviction == 09:39:37 (1789133977) [11253.920463] Lustre: DEBUG MARKER: == sanity test 124f: LFRU priv threshold inc/dec adjustment ========================================================== 09:40:26 (1789134026) [11352.699282] Lustre: DEBUG MARKER: == sanity test 124g: LFRU performance test =============== 09:42:06 (1789134126) [12329.845313] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 09:58:23 (1789135103) [12338.967066] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 09:58:33 (1789135113) [12346.345586] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 09:58:40 (1789135120) [12356.299759] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 09:58:49 (1789135129) [12366.295394] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [12403.741983] Lustre: DEBUG MARKER: == sanity test 127d: OSC RPC latency histograms for read and write latency ========================================================== 09:59:37 (1789135177) [12421.372636] Lustre: DEBUG MARKER: == sanity test 127e: client IO latency histograms by size ========================================================== 09:59:54 (1789135194) [12443.714064] Lustre: DEBUG MARKER: == sanity test 127f: OST IO latency histograms by size === 10:00:17 (1789135217) [12445.684399] Lustre: DEBUG MARKER: SKIP: sanity test_127f ldiskfs only [12447.925600] Lustre: DEBUG MARKER: == sanity test 127g: cached_read_bytes tracks page cache hits ========================================================== 10:00:21 (1789135221) [12463.371739] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 10:00:37 (1789135237) [12472.052396] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 10:00:46 (1789135246) [12473.932921] Lustre: DEBUG MARKER: SKIP: sanity test_129 ldiskfs only test [12475.801965] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 10:00:50 (1789135250) [12477.778867] Lustre: DEBUG MARKER: SKIP: sanity test_130a LU-1941: FIEMAP unimplemented on ZFS [12479.472853] Lustre: DEBUG MARKER: SKIP: sanity test_130b skipping ALWAYS excluded test 130b [12481.671539] Lustre: DEBUG MARKER: SKIP: sanity test_130c skipping ALWAYS excluded test 130c [12483.501784] Lustre: DEBUG MARKER: SKIP: sanity test_130d skipping ALWAYS excluded test 130d [12485.995510] Lustre: DEBUG MARKER: SKIP: sanity test_130e skipping ALWAYS excluded test 130e [12488.302947] Lustre: DEBUG MARKER: SKIP: sanity test_130f skipping ALWAYS excluded test 130f [12490.768818] Lustre: DEBUG MARKER: SKIP: sanity test_130g skipping ALWAYS excluded test 130g [12493.253275] Lustre: DEBUG MARKER: == sanity test 130h: FIEMAP deadlock ===================== 10:01:06 (1789135266) [12509.552806] Lustre: DEBUG MARKER: == sanity test 130i: FIEMAP (DoM file) =================== 10:01:23 (1789135283) [12511.584577] Lustre: DEBUG MARKER: SKIP: sanity test_130i LU-1941: FIEMAP unimplemented on ZFS [12514.185687] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 10:01:27 (1789135287) [12522.974814] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 10:01:37 (1789135297) [12530.887907] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 10:01:45 (1789135305) [12537.769471] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 10:01:52 (1789135312) [12544.905328] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 10:01:59 (1789135319) [12553.265925] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 10:02:07 (1789135327) [12577.885483] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 10:02:31 (1789135351) [12604.056666] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 10:02:58 (1789135378) [12660.432597] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 10:03:53 (1789135433) [12703.499298] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 10:04:37 (1789135477) [12718.753531] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 10:04:52 (1789135492) [12742.412096] Lustre: server umount lustre-MDT0000 complete [12747.064804] LustreError: 78269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135522 with bad export cookie 2363528317212361415 [12747.080881] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12756.596819] Lustre: server umount lustre-OST0000 complete [12760.544218] Lustre: 3312:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789135520/real 1789135520] req@ffffa03584c94780 x1876031388177536/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789135536 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12760.602670] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12763.173966] Lustre: server umount lustre-OST0001 complete [12784.044423] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing unload_modules_local [12788.634469] Key type lgssc unregistered [12788.949756] LNet: 101630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12788.966153] LNetError: 101630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12789.007254] LNet: Removed LNI 192.168.203.140@tcp [12789.951155] Key type .llcrypt unregistered [12789.957133] Key type ._llcrypt unregistered [12810.279809] Key type ._llcrypt registered [12810.283322] Key type .llcrypt registered [12810.401892] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [12812.011106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12812.055686] alg: No test for adler32 (adler32-zlib) [12813.128900] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [12813.295740] LNet: Added LNI 192.168.203.140@tcp [8/256/0/180] [12814.937140] Key type lgssc registered [12816.569865] Lustre: Echo OBD driver; http://www.lustre.org/ [12833.349613] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [12845.339723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12847.502305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12852.126448] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [12855.293028] Lustre: 103858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12861.990535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12870.323167] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [12872.148974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000402:45222 to 0x240000402:45569) [12877.323306] LustreError: 104217:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12877.368614] LustreError: 104217:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12880.257701] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12885.505880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:28037 to 0x280000402:28065) [12887.268563] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [12898.222804] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12902.184267] Lustre: 105769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12914.006478] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 10:08:07 (1789135687) [12923.027217] LNet: 106261:0:(debug.c:375:cfs_str2mask()) unknown mask ''. [12923.027217] mask usage: [+|-] ... [12923.574387] LNet: 106303:0:(debug.c:375:cfs_str2mask()) unknown mask ''. [12923.574387] mask usage: [+|-] ... [12923.590426] LNet: 106303:0:(debug.c:375:cfs_str2mask()) Skipped 3 previous similar messages [12923.642767] Lustre: DEBUG MARKER:  [12924.577176] LNet: 106384:0:(debug.c:375:cfs_str2mask()) unknown mask ''. [12924.577176] mask usage: [+|-] ... [12924.600802] LNet: 106384:0:(debug.c:375:cfs_str2mask()) Skipped 2 previous similar messages [12961.659707] LNet: 107440:0:(debug.c:375:cfs_str2mask()) unknown mask ''. [12961.659707] mask usage: [+|-] ... [12962.813654] Lustre: DEBUG MARKER:  [12962.820589] Lustre: DEBUG MARKER:  [13010.959208] Lustre: server umount lustre-MDT0000 complete [13014.889242] LustreError: 103409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135790 with bad export cookie 12644240017633473827 [13014.904397] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13025.406414] Lustre: server umount lustre-OST0000 complete [13039.640610] Lustre: server umount lustre-OST0001 complete [13053.343156] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing unload_modules_local [13056.446796] Key type lgssc unregistered [13056.734826] LNet: 109540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13056.739993] LNetError: 109540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13056.770357] LNet: Removed LNI 192.168.203.140@tcp [13057.615162] Key type .llcrypt unregistered [13057.623697] Key type ._llcrypt unregistered [13075.177834] Key type ._llcrypt registered [13075.180978] Key type .llcrypt registered [13075.327891] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [13076.637726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13076.677245] alg: No test for adler32 (adler32-zlib) [13078.327986] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [13078.898357] LNet: Added LNI 192.168.203.140@tcp [8/256/0/180] [13080.783253] Key type lgssc registered [13082.633977] Lustre: Echo OBD driver; http://www.lustre.org/ [13104.585922] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [13114.325062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13116.613868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13122.392703] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [13125.235734] Lustre: 111778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [13132.036652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13138.990308] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [13141.255200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000402:45222 to 0x240000402:45601) [13147.025247] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13152.285471] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:28037 to 0x280000402:28097) [13153.140443] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [13162.434442] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13165.995285] Lustre: 113687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [13172.981726] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 10:12:27 (1789135947) [14196.398330] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 10:29:31 (1789136971) [14214.077073] Lustre: *** cfs_fail_loc=327, val=500*** [14215.171963] Lustre: *** cfs_fail_loc=327, val=500*** [14215.184142] Lustre: Skipped 512 previous similar messages [14248.936920] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 10:30:23 (1789137023) [14255.783782] Lustre: *** cfs_fail_loc=328, val=500*** [14257.785637] Lustre: *** cfs_fail_loc=328, val=500*** [14257.788021] Lustre: Skipped 145 previous similar messages [14261.787725] Lustre: *** cfs_fail_loc=328, val=500*** [14261.795309] Lustre: Skipped 349 previous similar messages [14261.870120] Lustre: 111345:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffffa035de0e92c0 x1876044900641792/t0(0) o101->e9c985d8-3674-44cc-80e9-01a57c999ab1@192.168.203.40@tcp:713/0 lens 648/0 e 0 to 0 dl 1789137048 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [14262.928222] Lustre: 111345:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffffa035de0e9e00 x1876044900642816/t0(0) o101->e9c985d8-3674-44cc-80e9-01a57c999ab1@192.168.203.40@tcp:714/0 lens 648/0 e 0 to 0 dl 1789137049 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [14264.016144] Lustre: 113901:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffffa035d34ded00 x1876044900643584/t0(0) o101->e9c985d8-3674-44cc-80e9-01a57c999ab1@192.168.203.40@tcp:715/0 lens 648/0 e 0 to 0 dl 1789137050 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [14266.205063] Lustre: 111346:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffffa03688322580 x1876044900645120/t0(0) o101->e9c985d8-3674-44cc-80e9-01a57c999ab1@192.168.203.40@tcp:717/0 lens 648/0 e 0 to 0 dl 1789137052 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [14266.234230] Lustre: 111346:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) Skipped 1 previous similar message [14270.475897] Lustre: *** cfs_fail_loc=328, val=500*** [14270.479734] Lustre: Skipped 12 previous similar messages [14270.483771] Lustre: 113901:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffffa035d34dc000 x1876044900648448/t0(0) o101->e9c985d8-3674-44cc-80e9-01a57c999ab1@192.168.203.40@tcp:722/0 lens 648/0 e 0 to 0 dl 1789137057 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [14270.509628] Lustre: 113901:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) Skipped 3 previous similar messages [14275.870358] LustreError: 170869:0:(ldlm_lockd.c:3246:lock_reclaim_threshold_mb_store()) Failed to set lock_reclaim_threshold_mb, rc = -22. [14292.476369] Lustre: DEBUG MARKER: == sanity test 134c: Lock memory accounting includes associated structures ========================================================== 10:31:06 (1789137066) [14309.400933] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [14311.379569] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [14313.704640] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 10:31:27 (1789137087) [14354.528170] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 10:32:08 (1789137128) [14406.380593] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 10:33:00 (1789137180) [14407.782665] Lustre: DEBUG MARKER: SKIP: sanity test_150b need >= 2.13.57 and ldiskfs for fallocate [14409.389854] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 10:33:03 (1789137183) [14410.702481] Lustre: DEBUG MARKER: SKIP: sanity test_150bb need >= 2.13.57 and ldiskfs for fallocate [14412.246537] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 10:33:06 (1789137186) [14413.612362] Lustre: DEBUG MARKER: SKIP: sanity test_150c need >= 2.13.57 and ldiskfs for fallocate [14415.089397] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 10:33:09 (1789137189) [14416.318862] Lustre: DEBUG MARKER: SKIP: sanity test_150d need >= 2.13.57 and ldiskfs for fallocate [14417.874905] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 10:33:12 (1789137192) [14419.447682] Lustre: DEBUG MARKER: SKIP: sanity test_150e need >= 2.13.57 and ldiskfs for fallocate [14421.157755] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 10:33:15 (1789137195) [14422.737796] Lustre: DEBUG MARKER: SKIP: sanity test_150f LU-14160: punch mode is not implemented on OSD ZFS [14424.436157] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 10:33:18 (1789137198) [14425.758264] Lustre: DEBUG MARKER: SKIP: sanity test_150g LU-14160: punch mode is not implemented on OSD ZFS [14427.389617] Lustre: DEBUG MARKER: == sanity test 150h: Verify extend fallocate updates the file size ========================================================== 10:33:21 (1789137201) [14428.597299] Lustre: DEBUG MARKER: SKIP: sanity test_150h need >= 2.13.57 and ldiskfs for fallocate [14430.064853] Lustre: DEBUG MARKER: == sanity test 150ia: Verify fallocate zero-range ZERO functionality ========================================================== 10:33:24 (1789137204) [14431.773967] Lustre: DEBUG MARKER: SKIP: sanity test_150ia zero-range mode is not implemented on OSD ZFS [14433.330685] Lustre: DEBUG MARKER: == sanity test 150ib: Verify fallocate zero-range PREALLOC functionality ========================================================== 10:33:27 (1789137207) [14434.891306] Lustre: DEBUG MARKER: SKIP: sanity test_150ib zero-range mode is not implemented on OSD ZFS [14436.816936] Lustre: DEBUG MARKER: == sanity test 150ic: Verify fallocate LARGE zero PREALLOC functionality ========================================================== 10:33:31 (1789137211) [14438.385553] Lustre: DEBUG MARKER: SKIP: sanity test_150ic zero-range mode is not implemented on OSD ZFS [14440.086326] Lustre: DEBUG MARKER: == sanity test 150id: fallocate that fails must not leave a dirty page behind ========================================================== 10:33:34 (1789137214) [14441.538127] Lustre: DEBUG MARKER: SKIP: sanity test_150id fallocate zero-range is ldiskfs only [14443.827067] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 10:33:38 (1789137218) [14446.976755] Lustre: DEBUG MARKER: SKIP: sanity test_151 not cache-capable obdfilter [14448.558196] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 10:33:43 (1789137223) [14455.804771] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 10:33:50 (1789137230) [14461.747307] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 10:33:56 (1789137236) [14467.849255] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 10:34:02 (1789137242) [14474.071314] Lustre: DEBUG MARKER: == sanity test 154C: lfs fid2path on OST FID ============= 10:34:08 (1789137248) [14503.533938] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 10:34:37 (1789137277) [14505.321634] LustreError: 111344:0:(fld_handler.c:268:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [14513.281286] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 10:34:47 (1789137287) [14514.691053] Lustre: DEBUG MARKER: SKIP: sanity test_154b needs >= 2 MDTs [14516.155711] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 10:34:50 (1789137290) [14522.896218] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 10:34:57 (1789137297) [14530.348790] Lustre: DEBUG MARKER: == sanity test 154db: fid is stored in dir entries ======= 10:35:04 (1789137304) [14531.844616] Lustre: DEBUG MARKER: SKIP: sanity test_154db ldiskfs only test [14533.802337] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 10:35:07 (1789137307) [14540.396569] Lustre: DEBUG MARKER: == sanity test 154ea: .lustre is not returned by readdir (2) ========================================================== 10:35:14 (1789137314) [14596.428725] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 10:36:10 (1789137370) [14603.825350] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 10:36:18 (1789137378) [15780.742198] Lustre: DEBUG MARKER: == sanity test 154h: Verify interactive path2fid ========= 10:55:54 (1789138554) [15787.639692] Lustre: DEBUG MARKER: == sanity test 154i: fid2path for path longer than PATH_MAX ========================================================== 10:56:01 (1789138561) [15829.630924] Lustre: DEBUG MARKER: == sanity test 154j: fid2path for long path crossing MDT boundary ========================================================== 10:56:43 (1789138603) [15831.234436] Lustre: DEBUG MARKER: SKIP: sanity test_154j needs >= 2 MDTs [15832.897664] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 10:56:47 (1789138607) [15845.791794] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 10:57:00 (1789138620) [15858.304560] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 10:57:12 (1789138632) [15870.787517] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 10:57:25 (1789138645) [15883.957806] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 10:57:38 (1789138658) [15946.565703] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 10:58:40 (1789138720) [16007.550752] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 10:59:41 (1789138781) [16068.342764] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 11:00:42 (1789138842) [16130.445342] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 11:01:44 (1789138904) [16132.933684] Lustre: DEBUG MARKER: SKIP: sanity test_156 LU-1956/LU-2261: stats not implemented on OSD ZFS [16135.722498] Lustre: DEBUG MARKER: == sanity test 157a: llapi pool pinning API tests ======== 11:01:49 (1789138909) [16145.679896] Lustre: DEBUG MARKER: == sanity test 157b: lustre.pin inheritance on create ==== 11:01:59 (1789138919) [16154.463283] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 11:02:08 (1789138928) [16157.585147] Lustre: lustre-MDD0000: changelog on [16175.334485] Lustre: Failing over lustre-MDT0000 [16175.980917] Lustre: server umount lustre-MDT0000 complete [16185.048555] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16185.327701] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.40@tcp (not set up) [16185.454417] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16185.636407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16185.702304] Lustre: lustre-MDD0000: changelog on [16185.717380] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [16187.057277] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [16188.684101] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [16188.738511] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:28930 to 0x280000402:28961) [16188.740051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000402:46430 to 0x240000402:46465) [16190.000488] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [16190.956524] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16194.016083] Lustre: 110027:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789138953/real 1789138953] req@ffffa035c8000780 x1876044918250368/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789138969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [16194.350331] Lustre: lustre-MDD0000: changelog off [16199.137561] Lustre: 110025:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789138958/real 1789138958] req@ffffa0368b07a1c0 x1876044918250752/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789138974 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [16199.179563] Lustre: 110025:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16205.928918] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 11:03:00 (1789138980) [16208.682661] Lustre: lustre-MDD0000: changelog on [16217.970387] Lustre: lustre-MDD0000: changelog off [16220.837847] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 11:03:15 (1789138995) [16224.008744] Lustre: lustre-MDD0000: changelog on [16234.784681] Lustre: lustre-MDD0000: changelog off [16237.788718] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 11:03:31 (1789139011) [16240.221125] Lustre: DEBUG MARKER: SKIP: sanity test_160d needs >= 2 MDTs [16242.026248] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 11:03:36 (1789139016) [16245.301736] Lustre: lustre-MDD0000: changelog on [16254.240340] Lustre: lustre-MDD0000: changelog off [16256.945738] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 11:03:51 (1789139031) [16260.279846] Lustre: lustre-MDD0000: changelog on [16264.597512] Lustre: DEBUG MARKER: 1789139038: creating first dirs [16283.331504] Lustre: *** cfs_fail_loc=1313, val=3*** [16283.334218] Lustre: 185579:0:(mdd_dir.c:995:mdd_changelog_emrg_cleanup()) lustre-MDD0000: changelog has only 3 free catalog entries [16283.346925] Lustre: 185579:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [16283.355880] Lustre: 189145:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl6 idle for 21s with 4 unprocessed records [16304.129230] Lustre: lustre-MDD0000: changelog off [16307.450541] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 11:04:41 (1789139081) [16310.368785] Lustre: lustre-MDD0000: changelog on [16324.617104] Lustre: 185579:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [16324.643643] Lustre: 190744:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl8 idle for 12s with 4 unprocessed records [16340.407300] Lustre: lustre-MDD0000: changelog off [16343.548842] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 11:05:17 (1789139117) [16346.983302] Lustre: lustre-MDD0000: changelog on [16372.800887] Lustre: *** cfs_fail_loc=1316, val=0*** [16372.805335] Lustre: 185382:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0000: simulate starting changelog garbage collection [16372.840399] Lustre: 192342:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl10 idle for 23s with 4 unprocessed records [16375.329241] Lustre: Failing over lustre-MDT0000 [16379.891081] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.40@tcp (stopping) [16380.389915] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16380.405468] Lustre: Skipped 2 previous similar messages [16382.836284] Lustre: server umount lustre-MDT0000 complete [16393.035029] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16393.995887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16394.151546] Lustre: 193300:0:(mdd_device.c:628:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 34 to index 35, being cleared now [16394.181851] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [16399.361960] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [16400.455865] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16400.468763] Lustre: Skipped 1 previous similar message [16401.025903] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [16401.755941] Lustre: lustre-MDT0000: Recovery over after 0:02, of 1 clients 1 recovered and 0 were evicted. [16401.814338] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:28963 to 0x280000402:28993) [16401.817670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000402:46430 to 0x240000402:46497) [16420.150917] Lustre: lustre-MDD0000: changelog off [16422.596772] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 11:06:37 (1789139197) [16425.241853] Lustre: lustre-MDD0000: changelog on [16425.244390] Lustre: Skipped 1 previous similar message [16430.179683] LustreError: 194775:0:(mdd_device.c:2067:mdd_changelog_user_purge()) cfs_race id 1315 sleeping [16432.925482] LustreError: 194910:0:(mdd_device.c:1782:mdd_changelog_user_register()) cfs_fail_race id 1315 waking [16432.934306] LustreError: 194775:0:(mdd_device.c:2067:mdd_changelog_user_purge()) cfs_fail_race id 1315 awake: rc=2265 [16448.490808] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 11:07:02 (1789139222) [16463.914080] Lustre: lustre-MDD0000: changelog off [16463.922287] Lustre: Skipped 2 previous similar messages [16468.608680] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 11:07:22 (1789139242) [16474.521138] LustreError: 193311:0:(mdd_dir.c:1050:mdd_changelog_store()) cfs_fail_timeout id 15d sleeping for 3000ms [16477.543139] LustreError: 193311:0:(mdd_dir.c:1050:mdd_changelog_store()) cfs_fail_timeout id 15d awake [16495.123919] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 11:07:48 (1789139268) [16516.430581] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 11:08:10 (1789139290) [16527.635875] LustreError: 193620:0:(mdd_device.c:400:llog_changelog_cancel_cb()) cfs_race id 15f sleeping [16529.647325] LustreError: 193311:0:(mdd_device.c:400:llog_changelog_cancel_cb()) cfs_fail_race id 15f waking [16529.669369] LustreError: 193620:0:(mdd_device.c:400:llog_changelog_cancel_cb()) cfs_fail_race id 15f awake: rc=2983 [16541.172836] Lustre: lustre-MDD0000: changelog off [16541.177365] Lustre: Skipped 2 previous similar messages [16544.367869] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 11:08:38 (1789139318) [18611.084772] LustreError: 193310:0:(mdd_device.c:414:llog_changelog_cancel_cb()) cfs_race id 16c sleeping [18613.078417] LustreError: 198728:0:(llog_osd.c:1130:llog_osd_next_block()) cfs_fail_race id 16c waking [18613.095073] LustreError: 193310:0:(mdd_device.c:414:llog_changelog_cancel_cb()) cfs_fail_race id 16c awake: rc=3003 [18624.812079] Lustre: lustre-MDD0000: changelog off [18627.501807] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 11:43:21 (1789141401) [18630.307412] Lustre: lustre-MDD0000: changelog on [18630.315904] Lustre: Skipped 6 previous similar messages [18631.362105] LustreError: 200839:0:(mdd_device.c:1736:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [18632.158894] Lustre: 200888:0:(mdd_device.c:1753:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [18632.954807] LustreError: 200936:0:(mdd_device.c:1745:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [18657.969486] Lustre: lustre-MDD0000: changelog off [18663.214899] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 11:43:57 (1789141437) [18665.387764] Lustre: DEBUG MARKER: SKIP: sanity test_160p ldiskfs only test [18668.014433] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 11:44:01 (1789141441) [18670.585662] Lustre: lustre-MDD0000: changelog on [18680.474125] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records * time ========================================================== 11:44:14 (1789141454) [18695.509549] Lustre: 193311:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [18695.521915] Lustre: 203092:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl24 idle for 864011s with 500000004 unprocessed records [18695.541985] Lustre: lustre-MDD0000: changelog off [18695.543712] Lustre: Skipped 1 previous similar message [18711.709489] Lustre: DEBUG MARKER: == sanity test 160t: changelog garbage collect on lack of space ========================================================== 11:44:45 (1789141485) [18714.936140] Lustre: lustre-MDD0000: changelog on [18714.937808] Lustre: Skipped 1 previous similar message [18774.490330] Lustre: *** cfs_fail_loc=18c, val=1211660*** [18774.494545] Lustre: 198728:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [18774.521670] Lustre: 204694:0:(mdd_dir.c:966:mdd_changelog_is_space_safe()) lustre-MDD0000: changelog size 1MB with 1MB space limit [18774.554189] Lustre: 204694:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl25-user1 idle for 60s with 7507 unprocessed records [18788.790154] Lustre: lustre-MDD0000: changelog off [18821.118854] Lustre: DEBUG MARKER: == sanity test 160u: changelog rename record type name and sname strings are correct ========================================================== 11:46:35 (1789141595) [18824.521505] Lustre: lustre-MDD0000: changelog on [18835.237340] Lustre: DEBUG MARKER: == sanity test 160v: setattr preserves mtime update despite inter-client clock skew ========================================================== 11:46:49 (1789141609) [18836.732268] Lustre: DEBUG MARKER: SKIP: sanity test_160v need >= 2 clients [18838.648840] Lustre: DEBUG MARKER: == sanity test 160w: lfs changelog --user --mask ========= 11:46:52 (1789141612) [18859.434654] Lustre: DEBUG MARKER: == sanity test 160x: changelog users do not disappear ==== 11:47:13 (1789141633) [18909.357292] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 11:48:03 (1789141683) [18952.890428] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 11:48:47 (1789141727) [18954.436951] Lustre: DEBUG MARKER: SKIP: sanity test_161b skipping remote directory test [18956.223741] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 11:48:50 (1789141730) [18959.604870] Lustre: lustre-MDD0000: changelog on [18959.608866] Lustre: Skipped 2 previous similar messages [18970.819780] Lustre: lustre-MDD0000: changelog off [18970.824458] Lustre: Skipped 3 previous similar messages [18973.507149] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 11:49:07 (1789141747) [18988.621927] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 11:49:22 (1789141762) [18997.565913] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 11:49:31 (1789141771) [18999.161763] Lustre: DEBUG MARKER: SKIP: sanity test_162b needs >= 2 MDTs [19001.051970] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 11:49:35 (1789141775) [19065.442907] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 11:50:38 (1789141838) [19074.266671] Lustre: Failing over lustre-OST0000 [19074.497254] Lustre: server umount lustre-OST0000 complete [19074.534549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19074.545218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19074.561895] LustreError: 176994:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19074.575206] LustreError: 176994:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [19076.080677] LustreError: 176992:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19078.111412] LustreError: 176936:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [19081.192919] LustreError: 112145:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19086.308727] LustreError: 176992:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19086.340531] LustreError: 176992:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [19092.062362] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [19092.077553] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [19093.440896] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [19094.126873] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [19094.132945] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [19094.153254] Lustre: Skipped 1 previous similar message [19097.794684] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [19102.304543] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 11:51:15 (1789141875) [19135.484718] Lustre: Failing over lustre-OST0000 [19135.508609] LustreError: 213010:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x240000402:0xb5a3:0x0].0x0 (ffffa035a6e80200) refcount nonzero (2) after lock cleanup; forcing cleanup. [19137.691941] Lustre: server umount lustre-OST0000 complete [19138.021164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19138.043465] LustreError: 170125:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19138.060191] LustreError: 170125:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [19147.255243] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [19147.277980] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [19148.519573] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [19149.025884] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [19149.027409] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [19154.099435] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [19157.389618] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 11:52:11 (1789141931) [19186.803580] Lustre: Failing over lustre-OST0000 [19186.822103] LustreError: 214582:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x240000402:0xb5a3:0x0].0x0 (ffffa035c0e3df00) refcount nonzero (2) after lock cleanup; forcing cleanup. [19187.020358] Lustre: server umount lustre-OST0000 complete [19188.196566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19188.225343] LustreError: 176936:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19188.258933] LustreError: 176936:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [19196.134985] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [19196.156691] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [19197.932446] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [19198.141512] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [19198.141563] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [19202.545924] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [19206.209513] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 11:53:00 (1789141980) [19253.566714] Lustre: Failing over lustre-OST0000 [19253.599060] LustreError: 216980:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x240000402:0xb5a3:0x0].0x0 (ffffa03684e32a00) refcount nonzero (2) after lock cleanup; forcing cleanup. [19253.813352] Lustre: server umount lustre-OST0000 complete [19254.753744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19254.758490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19254.784681] LustreError: 170122:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19254.805804] LustreError: 170122:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [19262.659782] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [19262.671671] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [19263.859525] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [19263.978626] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [19263.979591] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [19269.744836] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [19273.844254] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 11:54:07 (1789142047) [19275.785826] Lustre: DEBUG MARKER: SKIP: sanity test_165e needs >= 2 MDTs [19277.996721] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 11:54:11 (1789142051) [19286.130569] Lustre: Failing over lustre-OST0000 [19286.158657] LustreError: 218247:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x240000402:0xb5a3:0x0].0x0 (ffffa03684e32d00) refcount nonzero (2) after lock cleanup; forcing cleanup. [19286.318350] Lustre: server umount lustre-OST0000 complete [19288.566613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19303.031899] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [19303.051292] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [19303.364385] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [19305.235396] Lustre: lustre-OST0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [19305.252638] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [19309.163750] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [19313.150505] Lustre: DEBUG MARKER: == sanity test 165g: ofd_access_log_reader --keepalive works ========================================================== 11:54:47 (1789142087) [19356.966667] Lustre: Failing over lustre-OST0000 [19357.019498] LustreError: 219835:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x240000402:0xb5a3:0x0].0x0 (ffffa036bcfd5f00) refcount nonzero (2) after lock cleanup; forcing cleanup. [19357.248499] Lustre: server umount lustre-OST0000 complete [19359.205115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19359.231730] LustreError: 112144:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19359.243519] LustreError: 112144:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [19378.291832] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [19378.306716] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [19379.615962] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [19379.816694] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [19379.820082] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [19385.324315] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug all all [19389.362855] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 11:56:03 (1789142163) [19391.429754] Lustre: DEBUG MARKER: creating a 10 Mb file [19467.510498] Lustre: DEBUG MARKER: starting reads [19469.504245] Lustre: DEBUG MARKER: truncating the file [19471.751703] Lustre: DEBUG MARKER: killing dd [19473.188457] Lustre: DEBUG MARKER: removing the temporary file [19481.273930] Lustre: DEBUG MARKER: == sanity test 170a: test lctl df to handle corrupted log ========================================================== 11:57:35 (1789142255) [19506.118405] Lustre: DEBUG MARKER: == sanity test 170b: check filename encoding ============= 11:58:00 (1789142280) [19538.632229] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 11:58:32 (1789142312) [19548.642782] Lustre: DEBUG MARKER: == sanity test 172: manual device removal with lctl cleanup/detach ================================================================ 11:58:42 (1789142322) [19558.655394] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 11:58:52 (1789142332) [19560.332533] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [19562.422839] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 11:58:56 (1789142336) [19566.553753] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_module obdecho/obdecho [19585.204320] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 11:59:19 (1789142359) [19589.371789] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_module obdecho/obdecho [19589.523530] Lustre: Echo OBD driver; http://www.lustre.org/ [19614.630566] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 11:59:48 (1789142388) [19715.494741] Lustre: DEBUG MARKER: == sanity test 182a: Test parallel modify metadata operations from mdc ========================================================== 12:01:29 (1789142489) [19825.405091] Lustre: DEBUG MARKER: == sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 12:03:19 (1789142599) [19827.654850] Lustre: DEBUG MARKER: SKIP: sanity test_182b needs >= 2 MDTs [19830.096995] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 12:03:24 (1789142604) [19831.676102] Lustre: *** cfs_fail_loc=148, val=0*** [19840.839328] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 12:03:34 (1789142614) [19852.385766] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 12:03:46 (1789142626) [19860.617241] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 12:03:54 (1789142634) [19912.007851] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 12:04:45 (1789142685) [19936.357477] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 12:05:10 (1789142710) [19956.598133] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 12:05:30 (1789142730) [19964.267600] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 12:05:38 (1789142738) [19974.237773] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 12:05:48 (1789142748) [19985.123817] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 12:05:59 (1789142759) [19995.812275] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 12:06:09 (1789142769) [20003.652439] Lustre: DEBUG MARKER: == sanity test 190a: check lfs project -p works with project name ========================================================== 12:06:17 (1789142777) [20012.685531] Lustre: DEBUG MARKER: == sanity test 190b: check lfs find --project works with project name ========================================================== 12:06:26 (1789142786) [20103.166337] Lustre: DEBUG MARKER: == sanity test 190c: check lfs project -p works with u:USERNAME ========================================================== 12:07:57 (1789142877) [20117.933491] Lustre: DEBUG MARKER: == sanity test complete, duration 19841 sec ============== 12:08:12 (1789142892) [20120.783502] Lustre: DEBUG MARKER: === sanity: start cleanup 12:08:15 (1789142895) === [20165.135641] Lustre: DEBUG MARKER: === sanity: finish cleanup 12:08:59 (1789142939) === [20166.699276] LustreError: 176945:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa0359619c780 x1876044923775872/t0(0) o104->lustre-OST0000@192.168.203.40@tcp:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 [20171.753251] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20171.758349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20171.761425] Lustre: Skipped 1 previous similar message [20171.781651] Lustre: Skipped 2 previous similar messages [20175.109346] Lustre: server umount lustre-MDT0000 complete [20179.441533] LustreError: 169895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142955 with bad export cookie 1331210720891486142 [20179.452394] LustreError: 169895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [20179.456766] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20179.693317] Lustre: server umount lustre-OST0000 complete [20183.741132] Lustre: server umount lustre-OST0001 complete [20195.590354] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing unload_modules_local [20198.145194] Key type lgssc unregistered [20198.392704] LNet: 234176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20198.406489] LNetError: 234176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20198.423347] LNet: Removed LNI 192.168.203.140@tcp [20199.233140] Key type .llcrypt unregistered [20199.235176] Key type ._llcrypt unregistered