[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] mapped at [ffffffffff2003f0] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5200 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1d87 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1c23 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1c97 00090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1d27 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1d5f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 419745758 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.522282] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.526830] pid_max: default: 32768 minimum: 301 [ 0.528764] Security Framework initialized [ 0.531635] SELinux: Initializing. [ 0.532992] SELinux: Starting in permissive mode [ 0.535687] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.541978] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.547798] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.554012] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.559249] Initializing cgroup subsys memory [ 0.563488] Initializing cgroup subsys devices [ 0.565654] Initializing cgroup subsys freezer [ 0.567663] Initializing cgroup subsys net_cls [ 0.568940] Initializing cgroup subsys blkio [ 0.570267] Initializing cgroup subsys perf_event [ 0.572831] Initializing cgroup subsys hugetlb [ 0.574970] Initializing cgroup subsys pids [ 0.576844] Initializing cgroup subsys net_prio [ 0.578781] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.582995] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.585544] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.587859] tlb_flushall_shift: 6 [ 0.589520] FEATURE SPEC_CTRL Present [ 0.591009] FEATURE IBPB_SUPPORT Present [ 0.592616] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.594762] Spectre V2 : Vulnerable [ 0.595827] Speculative Store Bypass: Vulnerable [ 0.598152] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.606028] ACPI: Core revision 20130517 [ 0.619259] ACPI: All ACPI Tables successfully acquired [ 0.629376] ftrace: allocating 30294 entries in 119 pages [ 0.699935] Enabling x2apic [ 0.700905] Enabled x2apic [ 0.702340] Switched APIC routing to physical x2apic. [ 0.706536] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.708768] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.712550] TSC deadline timer enabled [ 0.712800] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.720802] ... version: 2 [ 0.722328] ... bit width: 48 [ 0.723974] ... generic registers: 4 [ 0.726937] ... value mask: 0000ffffffffffff [ 0.728862] ... max period: 00007fffffffffff [ 0.731083] ... fixed-purpose events: 3 [ 0.736692] ... event mask: 000000070000000f [ 0.738544] KVM setup paravirtual spinlock [ 0.751810] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.755049] KVM setup async PF for cpu 1 [ 0.759534] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.767706] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.743760] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.780752] KVM setup async PF for cpu 2 [ 0.788689] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.800683] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.808956] Brought up 4 CPUs [ 0.809808] KVM setup async PF for cpu 3 [ 0.809815] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.812240] smpboot: Max logical packages: 1 [ 0.813479] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.824809] devtmpfs: initialized [ 0.826522] x86/mm: Memory block size: 128MB [ 0.832890] EVM: security.selinux [ 0.834904] EVM: security.ima [ 0.835812] EVM: security.capability [ 0.846487] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.850641] NET: Registered protocol family 16 [ 0.852608] cpuidle: using governor haltpoll [ 0.883037] ACPI: bus type PCI registered [ 0.886182] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.888928] PCI: Using configuration type 1 for base access [ 0.891084] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.902109] ACPI: Added _OSI(Module Device) [ 0.904578] ACPI: Added _OSI(Processor Device) [ 0.906076] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.907662] ACPI: Added _OSI(Processor Aggregator Device) [ 0.911070] ACPI: Added _OSI(Linux-Dell-Video) [ 0.913283] ACPI: EC: Look up EC in DSDT [ 0.916467] ACPI: Interpreter enabled [ 0.917891] ACPI: (supports S0 S3 S4 S5) [ 0.919298] ACPI: Using IOAPIC for interrupt routing [ 0.921141] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.924195] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.932069] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.934260] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.936851] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.939151] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.946915] acpiphp: Slot [2] registered [ 0.951285] acpiphp: Slot [5] registered [ 0.953513] acpiphp: Slot [6] registered [ 0.954821] acpiphp: Slot [7] registered [ 0.956717] acpiphp: Slot [8] registered [ 0.958252] acpiphp: Slot [9] registered [ 0.962673] acpiphp: Slot [10] registered [ 0.964990] acpiphp: Slot [3] registered [ 0.966774] acpiphp: Slot [4] registered [ 0.968339] acpiphp: Slot [11] registered [ 0.971791] acpiphp: Slot [12] registered [ 0.973537] acpiphp: Slot [13] registered [ 0.977162] acpiphp: Slot [14] registered [ 0.978934] acpiphp: Slot [15] registered [ 0.983053] acpiphp: Slot [16] registered [ 0.984450] acpiphp: Slot [17] registered [ 0.990616] acpiphp: Slot [18] registered [ 0.996872] acpiphp: Slot [19] registered [ 1.002430] acpiphp: Slot [20] registered [ 1.007836] acpiphp: Slot [21] registered [ 1.011908] acpiphp: Slot [22] registered [ 1.016239] acpiphp: Slot [23] registered [ 1.020013] acpiphp: Slot [24] registered [ 1.026028] acpiphp: Slot [25] registered [ 1.029738] acpiphp: Slot [26] registered [ 1.032950] acpiphp: Slot [27] registered [ 1.037742] acpiphp: Slot [28] registered [ 1.041284] acpiphp: Slot [29] registered [ 1.044186] acpiphp: Slot [30] registered [ 1.047596] acpiphp: Slot [31] registered [ 1.048941] PCI host bridge to bus 0000:00 [ 1.050381] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 1.052552] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 1.054798] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 1.057241] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 1.059880] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 1.062362] pci_bus 0000:00: root bus resource [bus 00-ff] [ 1.064262] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 1.065043] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 1.065957] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 1.072746] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 1.075784] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 1.079213] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 1.081745] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 1.084854] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 1.087164] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 1.087778] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 1.090764] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 1.093546] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 1.096663] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 1.106281] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 1.108228] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 1.109927] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 1.119252] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 1.143177] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 1.197785] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 1.220077] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.251616] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.272655] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.378624] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 1.394750] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.408059] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.427276] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.462065] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 1.477854] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.487330] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.497931] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.553479] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 1.603708] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.633347] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.646064] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.683314] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 1.693980] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.726504] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.732451] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.749368] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 1.762659] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.767381] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.769672] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.777641] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.781292] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.785193] vgaarb: loaded [ 1.787291] SCSI subsystem initialized [ 1.788379] ACPI: bus type USB registered [ 1.789854] usbcore: registered new interface driver usbfs [ 1.791826] usbcore: registered new interface driver hub [ 1.795873] usbcore: registered new device driver usb [ 1.801200] PCI: Using ACPI for IRQ routing [ 1.802901] PCI: pci_cache_line_size set to 64 bytes [ 1.803183] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.803188] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.803190] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.803889] NetLabel: Initializing [ 1.804920] NetLabel: domain hash size = 128 [ 1.806184] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.808140] NetLabel: unlabeled traffic allowed by default [ 1.814168] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.816120] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.823145] amd_nb: Cannot enumerate AMD northbridges [ 1.828352] Switched to clocksource kvm-clock [ 1.848791] pnp: PnP ACPI init [ 1.849927] ACPI: bus type PNP registered [ 1.851451] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.851532] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.851587] pnp 00:02: [dma 2] [ 1.851627] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.851702] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.851780] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.851858] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.852249] pnp: PnP ACPI: found 6 devices [ 1.853779] ACPI: bus type PNP unregistered [ 1.877295] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.877303] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.877307] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.877309] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.877311] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.877422] NET: Registered protocol family 2 [ 1.879742] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.881957] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.885065] TCP: Hash tables configured (established 32768 bind 32768) [ 1.887026] TCP: reno registered [ 1.888103] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.890476] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.893470] NET: Registered protocol family 1 [ 1.896527] RPC: Registered named UNIX socket transport module. [ 1.898623] RPC: Registered udp transport module. [ 1.899835] RPC: Registered tcp transport module. [ 1.901714] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.904956] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.907210] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.909400] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.911397] PCI: CLS 0 bytes, default 64 [ 1.912327] Unpacking initramfs... [ 3.852840] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 3.857886] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.860087] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 3.863407] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 3.866164] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 3.867985] RAPL PMU: hw unit of domain package 2^-0 Joules [ 3.869771] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 3.877507] cryptomgr_test (52) used greatest stack depth: 14480 bytes left [ 3.880345] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 3.882466] Initialise system trusted keyring [ 3.919048] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 3.921355] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 3.934500] zpool: loaded [ 3.937046] zbud: loaded [ 3.938416] VFS: Disk quotas dquot_6.6.0 [ 3.940032] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 3.944632] NFS: Registering the id_resolver key type [ 3.946374] Key type id_resolver registered [ 3.948384] Key type id_legacy registered [ 3.949666] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 3.952943] Key type big_key registered [ 3.954169] SELinux: Registering netfilter hooks [ 3.960815] cryptomgr_test (58) used greatest stack depth: 14048 bytes left [ 3.968881] cryptomgr_test (63) used greatest stack depth: 13600 bytes left [ 3.971863] NET: Registered protocol family 38 [ 3.973386] Key type asymmetric registered [ 3.974623] Asymmetric key parser 'x509' registered [ 3.976398] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 3.980091] io scheduler noop registered [ 3.981333] io scheduler deadline registered (default) [ 3.983163] io scheduler cfq registered [ 3.984558] io scheduler mq-deadline registered [ 3.985921] io scheduler kyber registered [ 3.990172] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 3.992119] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 3.995541] intel_idle: does not run on family 6 model 62 [ 3.995682] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.998134] ACPI: Power Button [PWRF] [ 4.000478] GHES: HEST is not enabled! [ 4.072022] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 4.140631] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 4.297358] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 4.371138] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 4.526422] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.554756] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 4.585728] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 4.590403] Non-volatile memory driver v1.3 [ 4.591730] Linux agpgart interface v0.103 [ 4.593238] crash memory driver: version 1.1 [ 4.594925] nbd: registered device at major 43 [ 4.603727] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 4.603769] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 4.603801] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 4.603838] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 4.603877] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 4.613531] virtio_blk virtio1: [vda] 53408 512-byte logical blocks (27.3 MB/26.0 MiB) [ 4.625775] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 4.625819] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 4.625858] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 4.625897] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 4.625942] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 4.640382] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.651487] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 4.651567] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 4.651609] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 4.651674] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 4.651731] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 4.663759] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.677699] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 4.677741] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 4.677778] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 4.677814] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 4.677860] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 4.695008] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.704011] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 4.704065] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 4.704111] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 4.704162] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 4.704208] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 4.717895] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.724950] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 4.724996] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 4.725036] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 4.725076] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 4.725119] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 4.733025] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.742950] rdac: device handler registered [ 4.744839] hp_sw: device handler registered [ 4.746391] emc: device handler registered [ 4.748672] libphy: Fixed MDIO Bus: probed [ 4.759698] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 4.762604] ehci-pci: EHCI PCI platform driver [ 4.766593] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 4.768842] ohci-pci: OHCI PCI platform driver [ 4.770481] uhci_hcd: USB Universal Host Controller Interface driver [ 4.775736] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.781032] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.782991] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.784881] mousedev: PS/2 mouse device common for all mice [ 4.790089] rtc_cmos 00:05: RTC can wake from S4 [ 4.795169] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 4.795637] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.802376] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.808918] hidraw: raw HID events driver (C) Jiri Kosina [ 4.811251] usbcore: registered new interface driver usbhid [ 4.813014] usbhid: USB HID core driver [ 4.814325] drop_monitor: Initializing network drop monitor service [ 4.816350] Netfilter messages via NETLINK v0.30. [ 4.829868] TCP: cubic registered [ 4.830909] Initializing XFRM netlink socket [ 4.832733] NET: Registered protocol family 10 [ 4.842056] NET: Registered protocol family 17 [ 4.843751] Key type dns_resolver registered [ 4.849661] mce: Using 10 MCE banks [ 4.854072] Loading compiled-in X.509 certificates [ 4.859285] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 4.867529] registered taskstats version 1 [ 4.874041] modprobe (72) used greatest stack depth: 13456 bytes left [ 4.875364] tsc: Refined TSC clocksource calibration: 2399.964 MHz [ 4.884968] Key type trusted registered [ 4.897724] Key type encrypted registered [ 4.898890] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 4.903048] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 4.910210] rtc_cmos 00:05: setting system clock to 2025-01-17 16:14:44 UTC (1737130484) [ 4.915387] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 4.920941] Write protecting the kernel read-only data: 12288k [ 4.924000] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 4.930003] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 4.945701] random: systemd: uninitialized urandom read (16 bytes read) [ 4.954953] random: systemd: uninitialized urandom read (16 bytes read) [ 4.959589] random: systemd: uninitialized urandom read (16 bytes read) [ 4.965239] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 4.975116] systemd[1]: Detected virtualization kvm. [ 4.977892] systemd[1]: Detected architecture x86-64. [ 4.982592] systemd[1]: Running in initial RAM disk. [ 4.996810] systemd[1]: No hostname configured. [ 5.000953] systemd[1]: Set hostname to . [ 5.003680] random: systemd: uninitialized urandom read (16 bytes read) [ 5.006076] systemd[1]: Initializing machine ID from random generator. [ 5.132033] dracut-rootfs-g (86) used greatest stack depth: 13248 bytes left [ 5.137708] random: systemd: uninitialized urandom read (16 bytes read) [ 5.140149] random: systemd: uninitialized urandom read (16 bytes read) [ 5.142779] random: systemd: uninitialized urandom read (16 bytes read) [ 5.146098] random: systemd: uninitialized urandom read (16 bytes read) [ 5.151979] random: systemd: uninitialized urandom read (16 bytes read) [ 5.155394] random: systemd: uninitialized urandom read (16 bytes read) [ 5.173604] systemd[1]: Reached target Local File Systems. [ 5.181878] systemd[1]: Created slice Root Slice. [ 5.190095] systemd[1]: Listening on udev Kernel Socket. [ 5.203211] systemd[1]: Created slice System Slice. [ 5.215410] systemd[1]: Listening on udev Control Socket. [ 5.227709] systemd[1]: Listening on Journal Socket. [ 5.235019] systemd[1]: Reached target Sockets. [ 5.242905] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 5.258102] systemd[1]: Starting Journal Service... [ 5.268862] systemd[1]: Reached target Timers. [ 5.286235] systemd[1]: Reached target Slices. [ 5.306864] systemd[1]: Starting Load Kernel Modules... [ 5.323122] systemd[1]: Reached target Swap. [ 5.332145] systemd[1]: Starting Setup Virtual Console... [ 5.350225] systemd[1]: Starting dracut cmdline hook... [ 5.357970] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 5.384678] systemd[1]: Started Load Kernel Modules. [ 5.394813] systemd[1]: Started Setup Virtual Console. [ 5.411077] systemd[1]: Started Journal Service. [ 5.797730] random: fast init done [ 5.819467] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 6.367709] libata version 3.00 loaded. [ 6.421046] ata_piix 0000:00:01.1: version 2.13 [ 6.427447] scsi host0: ata_piix [ 6.442406] scsi host1: ata_piix [ 6.443827] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.451512] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 6.698704] ip (322) used greatest stack depth: 13080 bytes left [ 6.783302] ip (345) used greatest stack depth: 12336 bytes left [ 10.317844] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 10.974326] systemd-journald[102]: Received SIGTERM from PID 1 (systemd). [ 11.441502] SELinux: Disabled at runtime. [ 11.443645] SELinux: Unregistering netfilter hooks [ 11.584053] ip_tables: (C) 2000-2006 Netfilter Core Team [ 11.589027] systemd[1]: Inserted module 'ip_tables' [ 12.541030] systemd-journald[569]: Received request to flush runtime journal from PID 1 [ 12.806288] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 12.933078] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.942201] cryptd: max_cpu_qlen set to 1000 [ 13.147145] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 13.168570] AVX version of gcm_enc/dec engaged. [ 13.170923] AES CTR mode by8 optimization enabled [ 13.194643] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 13.222678] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 13.240324] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 13.483144] EDAC MC: Ver: 3.0.0 [ 13.505399] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 13.505411] EDAC sbridge: Ver: 1.1.2 [ 18.872069] mount.nfs (778) used greatest stack depth: 10704 bytes left [ 32.080652] libcfs: loading out-of-tree module taints kernel. [ 32.082753] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 32.091428] alg: No test for adler32 (adler32-zlib) [ 32.870342] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 38.105871] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 38.318631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 38.455986] Lustre: Lustre: Build Version: 2.15.6_1_g67c9076 [ 38.622315] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 38.624334] LNet: Accept secure, port 988 [ 40.170369] Key type lgssc registered [ 40.476348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 40.886640] icp: module license 'CDDL' taints kernel. [ 40.888649] Disabling lock debugging due to kernel taint [ 42.508946] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 45.202986] vdc: vdc1 vdc9 [ 48.503211] vde: vde1 vde9 [ 52.154133] vdf: vdf1 vdf9 [ 57.799188] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 61.651890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 61.718180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 61.756507] Lustre: lustre-MDT0000: new disk, initializing [ 61.873952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 61.900149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 61.973606] mount.lustre (6582) used greatest stack depth: 9776 bytes left [ 63.022923] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 66.362923] Lustre: lustre-OST0000: new disk, initializing [ 66.365372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 66.368235] Lustre: Skipped 1 previous similar message [ 66.389341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 67.595484] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 70.765141] random: crng init done [ 70.915313] Lustre: lustre-OST0001: new disk, initializing [ 70.917138] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 70.939693] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 72.072141] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 77.041800] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 84.506600] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 90.658453] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing check_logdir /tmp/testlogs/ [ 92.267546] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing yml_node [ 93.488452] Lustre: DEBUG MARKER: Client: 2.15.6.1 [ 94.375134] Lustre: DEBUG MARKER: MDS: 2.15.6.1 [ 95.120541] Lustre: DEBUG MARKER: OSS: 2.15.6.1 [ 95.581530] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Fri Jan 17 11:16:13 EST 2025 [ 97.171214] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42b 42c 407 118c 118d 130b 130c 130d 130e 130f 130g [ 97.544657] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 115 135 136 230d 300o 51b [ 99.006499] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing check_config_client /mnt/lustre [ 102.997662] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 103.958848] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 104.918017] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 106.695772] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 11:16:24 (1737130584) [ 107.809983] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 108.456806] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 11:16:26 (1737130586) [ 111.608882] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 11:16:29 (1737130589) [ 165.006030] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 11:17:23 (1737130643) [ 167.285080] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 11:17:25 (1737130645) [ 167.736520] Lustre: *** cfs_fail_loc=15b, val=0*** [ 167.738242] Lustre: *** cfs_fail_loc=15b, val=0*** [ 169.653601] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 11:17:27 (1737130647) [ 171.883170] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 11:17:29 (1737130649) [ 172.233294] Lustre: *** cfs_fail_loc=19a, val=0*** [ 172.875112] Lustre: *** cfs_fail_loc=19a, val=0*** [ 172.879876] Lustre: Skipped 1 previous similar message [ 174.013682] Lustre: *** cfs_fail_loc=19a, val=0*** [ 174.015312] Lustre: Skipped 2 previous similar messages [ 176.095601] Lustre: *** cfs_fail_loc=19a, val=0*** [ 176.097732] Lustre: Skipped 5 previous similar messages [ 180.458200] Lustre: *** cfs_fail_loc=19a, val=0*** [ 180.461319] Lustre: Skipped 12 previous similar messages [ 188.705696] Lustre: *** cfs_fail_loc=19a, val=0*** [ 188.709242] Lustre: Skipped 21 previous similar messages [ 204.923810] Lustre: *** cfs_fail_loc=19a, val=0*** [ 204.926785] Lustre: Skipped 43 previous similar messages [ 215.242765] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 11:18:13 (1737130693) [ 215.843909] Lustre: DEBUG MARKER: SKIP: sanity test_60h Need at least 2 MDTs [ 216.498158] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 217.136106] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 11:18:15 (1737130695) [ 219.606346] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 11:18:17 (1737130697) [ 221.806245] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 11:18:19 (1737130699) [ 284.719332] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 11:19:22 (1737130762) [ 290.162347] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 11:19:28 (1737130768) [ 292.853034] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 293.513254] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 11:19:31 (1737130771) [ 296.201002] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 11:19:34 (1737130774) [ 321.729431] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 11:19:59 (1737130799) [ 322.666762] Lustre: *** cfs_fail_loc=725, val=0*** [ 323.793723] Lustre: *** cfs_fail_loc=725, val=0*** [ 326.311976] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 11:20:04 (1737130804) [ 329.681892] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 11:20:07 (1737130807) [ 398.359988] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 11:21:16 (1737130876) [ 409.197125] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 11:21:27 (1737130887) [ 411.306945] Lustre: *** cfs_fail_loc=513, val=0*** [ 411.308820] LustreError: 8687:0:(service.c:2115:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1821513366476608 [ 411.867030] Lustre: Failing over lustre-OST0000 [ 411.939223] Lustre: server umount lustre-OST0000 complete [ 412.428003] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 414.423813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 414.430965] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 417.432966] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 422.438112] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 422.446626] LustreError: Skipped 1 previous similar message [ 424.242621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.480351] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 425.636250] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 426.447827] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.138@tcp (at 0@lo) [ 426.447829] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 426.461968] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2513 to 0x0:2529 [ 429.023468] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 429.564408] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 433.041460] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 11:21:51 (1737130911) [ 434.875017] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 11:21:53 (1737130913) [ 436.626367] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 11:21:54 (1737130914) [ 438.302560] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 11:21:56 (1737130916) [ 439.955221] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 11:21:58 (1737130918) [ 441.605963] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 11:21:59 (1737130919) [ 443.262947] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 11:22:01 (1737130921) [ 444.956891] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 11:22:03 (1737130923) [ 446.712334] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 11:22:04 (1737130924) [ 448.800836] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 11:22:07 (1737130927) [ 451.500430] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 11:22:09 (1737130929) [ 452.067755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.072579] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 452.075633] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.138@tcp (at 0@lo) [ 452.075864] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2533 to 0x0:2561 [ 452.341159] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2509 to 0x0:2529 [ 456.971085] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 461.292774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.297159] Lustre: Skipped 1 previous similar message [ 461.298827] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 461.302035] Lustre: Skipped 1 previous similar message [ 461.303919] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 461.309274] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.138@tcp (at 0@lo) [ 461.309606] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3562 to 0x0:3585 [ 461.315382] Lustre: Skipped 1 previous similar message [ 462.293333] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 462.347572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 466.202212] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 470.141152] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.146138] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 470.149713] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 470.155576] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.201.138@tcp (at 0@lo) [ 470.156727] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3532 to 0x0:3553 [ 471.220248] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 471.278189] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 472.942529] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 11:22:31 (1737130951) [ 474.594434] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 11:22:32 (1737130952) [ 476.284872] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 11:22:34 (1737130954) [ 482.190405] Lustre: DEBUG MARKER: SKIP: sanity test_65n needs >= 2 MDTs [ 489.638630] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 11:22:47 (1737130967) [ 493.236164] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 11:22:51 (1737130971) [ 493.852680] Lustre: *** cfs_fail_loc=217, val=0*** [ 493.855421] Lustre: Skipped 2 previous similar messages [ 494.899407] Lustre: *** cfs_fail_loc=217, val=0*** [ 494.901411] Lustre: Skipped 5 previous similar messages [ 496.976550] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 497.439211] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 11:22:55 (1737130975) [ 499.329558] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 11:22:57 (1737130977) [ 501.323798] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 11:22:59 (1737130979) [ 529.298599] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 11:23:27 (1737131007) [ 531.140456] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 11:23:29 (1737131009) [ 533.217673] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 11:23:31 (1737131011) [ 535.325111] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 11:23:33 (1737131013) [ 550.721423] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 11:23:48 (1737131028) [ 560.566126] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 11:23:58 (1737131038) [ 562.594756] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 11:24:00 (1737131040) [ 562.680687] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.38@tcp inode [0x200000406:0xc25:0x0] object 0x0:3818 extent [0-1048575]: client csum d7295986, server csum d7295985 [ 564.306176] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 565.427558] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.38@tcp inode [0x200000406:0xc25:0x0] object 0x0:3818 extent [0-1048575], client returned csum aa7ffdea (type 1), server csum bc20b6e5 (type 1) [ 566.083928] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 567.182035] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.38@tcp inode [0x200000406:0xc25:0x0] object 0x0:3818 extent [0-1048575], client returned csum 9bca88ea (type 2), server csum dd7e8a43 (type 2) [ 567.885134] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 568.998389] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.38@tcp inode [0x200000406:0xc25:0x0] object 0x0:3818 extent [0-1048575], client returned csum 1dff7ade (type 4), server csum d7295985 (type 4) [ 569.689705] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 571.414308] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 572.496730] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.38@tcp inode [0x200000406:0xc25:0x0] object 0x0:3818 extent [0-1048575], client returned csum a7ba04b4 (type 20), server csum f68a035b (type 20) [ 572.504378] LustreError: Skipped 1 previous similar message [ 573.150559] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 574.842025] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 576.481548] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 578.138731] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 11:24:16 (1737131056) [ 580.153952] Lustre: 26673:0:(tgt_handler.c:1903:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000406:0xc26:0x0]:[0-1048575]-1dff7ade-d7295985 [ 580.161823] LustreError: dumping log to /tmp/lustre-log.1737131059.26673 [ 580.178914] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.38@tcp inode [0x200000406:0xc26:0x0] object 0x0:3853 extent [0-1048575], client returned csum 1dff7ade (type 4), server csum d7295985 (type 4) [ 580.186753] LustreError: Skipped 2 previous similar messages [ 590.077754] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 11:24:28 (1737131068) [ 590.211380] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.201.38@tcp inode [0x200000406:0xc28:0x0] object 0x0:3854 extent [0-1048575]: client csum b5ea7f3c, server csum b5ea7f3b [ 592.503512] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.38@tcp inode [0x200000406:0xc28:0x0] object 0x0:3854 extent [0-1048575], client returned csum fe0f685f (type 4), server csum b5ea7f3b (type 4) [ 594.153407] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 11:24:32 (1737131072) [ 594.565775] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 594.646605] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.38@tcp inode [0x200000406:0xc29:0x0] object 0x0:3819 extent [0-1048575]: client csum b2f1b12, server csum b2f1b11 [ 594.662779] LustreError: Skipped 5 previous similar messages [ 597.747752] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.38@tcp inode [0x200000406:0xc29:0x0] object 0x0:3819 extent [3145728-4194303]: client csum b2f1b12, server csum b2f1b11 [ 597.753238] LustreError: Skipped 8 previous similar messages [ 604.730252] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.38@tcp inode [0x200000406:0xc29:0x0] object 0x0:3819 extent [5242880-6291455]: client csum b2f1b12, server csum b2f1b11 [ 604.744602] LustreError: Skipped 20 previous similar messages [ 615.747655] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.38@tcp inode [0x200000406:0xc29:0x0] object 0x0:3819 extent [0-1048575]: client csum b2f1b12, server csum b2f1b11 [ 615.758897] LustreError: Skipped 12 previous similar messages [ 633.779045] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.38@tcp inode [0x200000406:0xc29:0x0] object 0x0:3819 extent [0-1048575]: client csum b2f1b12, server csum b2f1b11 [ 633.791338] LustreError: Skipped 23 previous similar messages [ 640.328634] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 667.829124] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.38@tcp inode [0x200000406:0xc29:0x0] object 0x0:3819 extent [3145728-4194303]: client csum 19eeae62, server csum 19eeae61 [ 667.841981] LustreError: Skipped 69 previous similar messages [ 686.340395] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 731.994449] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.38@tcp inode [0x200000406:0xc29:0x0] object 0x0:3819 extent [5242880-6291455]: client csum b5ea7f3c, server csum b5ea7f3b [ 732.009045] LustreError: Skipped 111 previous similar messages [ 732.533199] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 778.564041] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 824.526529] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 864.161875] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.38@tcp inode [0x200000406:0xc29:0x0] object 0x0:3819 extent [0-1048575]: client csum 887488b6, server csum 887488b5 [ 864.177987] LustreError: Skipped 255 previous similar messages [ 870.727024] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 916.861950] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 918.605232] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 11:29:56 (1737131396) [ 919.024128] Lustre: *** cfs_fail_loc=21a, val=0*** [ 920.488644] Lustre: *** cfs_fail_loc=21b, val=0*** [ 923.848439] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 11:30:02 (1737131402) [ 924.116625] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 924.701069] Lustre: Modifying parameter lustre.osc.lustre*.checksums in log params [ 926.356441] Lustre: Disabling parameter lustre.osc.lustre*.checksums in log params [ 928.715640] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 930.237902] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 11:30:08 (1737131408) [ 930.680567] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 931.131138] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 932.667871] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid 40 [ 948.298438] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid in IDLE state after 15 sec [ 949.768720] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid 40 [ 950.171004] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid in FULL state after 0 sec [ 950.584115] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 952.052527] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid 40 [ 972.796328] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid in IDLE state after 20 sec [ 974.416597] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid 40 [ 974.865031] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid in FULL state after 0 sec [ 975.309757] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 976.926683] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid 40 [ 997.788876] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid in IDLE state after 20 sec [ 999.577214] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid 40 [ 1000.036137] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid in FULL state after 0 sec [ 1000.561777] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1002.508873] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid 40 [ 1023.433081] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid in IDLE state after 20 sec [ 1025.083453] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid 40 [ 1025.503033] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid in FULL state after 0 sec [ 1025.999897] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1027.820667] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid 40 [ 1048.580975] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid in IDLE state after 20 sec [ 1050.081177] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid 40 [ 1050.459256] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid in FULL state after 0 sec [ 1050.865082] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1052.298963] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid 40 [ 1072.908577] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid in IDLE state after 20 sec [ 1074.260553] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid 40 [ 1074.619175] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid in FULL state after 0 sec [ 1075.017429] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1076.474890] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid 40 [ 1098.146499] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid in IDLE state after 21 sec [ 1099.532286] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid 40 [ 1099.890261] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8928800.ost_server_uuid in FULL state after 0 sec [ 1101.297865] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1101.677794] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 11:32:59 (1737131579) [ 1103.052696] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 11:33:01 (1737131581) [ 1103.458350] Lustre: DEBUG MARKER: SKIP: sanity test_77n f77n.sanity blocks not contiguous around hole [ 1103.852402] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 11:33:02 (1737131582) [ 1106.543918] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 11:33:04 (1737131584) [ 1108.660893] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ================================================================================= 11:33:06 (1737131586) [ 1130.299825] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 11:33:28 (1737131608) [ 1133.122890] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 11:33:31 (1737131611) [ 1133.420612] Lustre: *** cfs_fail_loc=228, val=0*** [ 1135.004660] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 11:33:33 (1737131613) [ 1135.308847] Lustre: *** cfs_fail_loc=228, val=0*** [ 1136.934001] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 11:33:35 (1737131615) [ 1142.063588] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ===================================================================== 11:33:40 (1737131620) [ 1144.116780] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 11:33:42 (1737131622) [ 1156.701103] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 11:33:54 (1737131634) [ 1159.945751] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 11:33:58 (1737131638) [ 1167.334953] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 11:34:05 (1737131645) [ 1210.374059] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 11:34:48 (1737131688) [ 1216.707109] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 11:34:54 (1737131694) [ 1218.740315] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 11:34:57 (1737131697) [ 1225.088527] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 11:35:03 (1737131703) [ 1227.011819] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 11:35:05 (1737131705) [ 1228.632947] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 11:35:06 (1737131706) [ 1234.454331] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 11:35:12 (1737131712) [ 1236.069625] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 11:35:14 (1737131714) [ 1238.336428] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 11:35:16 (1737131716) [ 1239.845383] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 11:35:18 (1737131718) [ 1242.664896] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 11:35:20 (1737131720) [ 1245.669591] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 11:35:23 (1737131723) [ 1246.065437] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 1246.430419] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 1246.790008] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 1247.163922] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 1248.648221] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 11:35:26 (1737131726) [ 1249.045043] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 1249.431298] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 1249.813946] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 1250.203641] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 1250.621500] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 1252.112465] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 11:35:30 (1737131730) [ 1253.580504] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 11:35:31 (1737131731) [ 1256.410667] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 11:35:34 (1737131734) [ 1258.051610] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 11:35:36 (1737131736) [ 1259.634982] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 11:35:37 (1737131737) [ 1261.383114] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 11:35:39 (1737131739) [ 1263.138561] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 11:35:41 (1737131741) [ 1264.645690] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 11:35:42 (1737131742) [ 1266.183002] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 11:35:44 (1737131744) [ 1267.690521] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 11:35:45 (1737131745) [ 1269.314061] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 11:35:47 (1737131747) [ 1270.963624] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 11:35:49 (1737131749) [ 1293.833913] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 11:36:12 (1737131772) [ 1308.078755] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 11:36:26 (1737131786) [ 1309.593818] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 11:36:27 (1737131787) [ 1512.324533] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 11:39:50 (1737131990) [ 1513.005523] Lustre: lustre-MDD0000: changelog on [ 1514.704429] Lustre: lustre-MDD0000: changelog off [ 1515.355945] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 11:39:53 (1737131993) [ 1515.457841] Lustre: lustre-OST0000: Client 27089d55-a1ce-4082-ae83-fff6e2fbd5b6 (at 192.168.201.38@tcp) reconnecting [ 1516.850526] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a8b28800.ost_server_uuid 40 [ 1517.213413] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800a8b28800.ost_server_uuid in FULL state after 0 sec [ 1518.798308] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 11:39:57 (1737131997) [ 1520.353533] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 11:39:58 (1737131998) [ 1527.342980] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 11:40:05 (1737132005) [ 1528.931742] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 11:40:07 (1737132007) [ 1530.593446] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 11:40:08 (1737132008) [ 1532.264239] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 11:40:10 (1737132010) [ 1543.855725] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 11:40:22 (1737132022) [ 1545.462073] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 11:40:23 (1737132023) [ 1547.065185] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 11:40:25 (1737132025) [ 1549.988981] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 11:40:28 (1737132028) [ 1551.624974] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [ 1552.042811] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 11:40:30 (1737132030) [ 1638.915697] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 11:41:57 (1737132117) [ 1639.622228] Lustre: *** cfs_fail_loc=147, val=0*** [ 1641.593317] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 11:41:59 (1737132119) [ 1643.128410] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 11:42:01 (1737132121) [ 1644.725124] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 11:42:02 (1737132122) [ 1645.132232] Lustre: *** cfs_fail_loc=217, val=0*** [ 1645.133664] Lustre: Skipped 3 previous similar messages [ 1646.971305] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 1647.361923] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 1647.749483] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 11:42:06 (1737132126) [ 1649.345718] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 11:42:07 (1737132127) [ 1651.071623] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 11:42:09 (1737132129) [ 1651.519465] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1652.538774] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1654.407495] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1657.412773] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1662.380831] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1664.240943] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 11:42:22 (1737132142) [ 1672.054656] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 11:42:30 (1737132150) [ 1672.477035] Lustre: *** cfs_fail_loc=220, val=0*** [ 1672.478723] Lustre: Skipped 3 previous similar messages [ 1674.383278] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 11:42:32 (1737132152) [ 1682.008352] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 11:42:40 (1737132160) [ 1683.588668] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 11:42:41 (1737132161) [ 1685.177168] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 11:42:43 (1737132163) [ 1689.445849] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 11:42:47 (1737132167) [ 1690.996035] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 11:42:49 (1737132169) [ 1692.577921] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 11:42:50 (1737132170) [ 1694.148067] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ========================================================== 11:42:52 (1737132172) [ 1695.569728] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [ 1699.159700] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 11:42:57 (1737132177) [ 1701.331859] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 11:42:59 (1737132179) [ 1703.488636] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 11:43:01 (1737132181) [ 1705.573920] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 11:43:03 (1737132183) [ 1707.672710] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 11:43:05 (1737132185) [ 1716.790277] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 11:43:15 (1737132195) [ 1726.013165] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 11:43:24 (1737132204) [ 1775.689948] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 11:44:13 (1737132253) [ 1777.339393] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 11:44:15 (1737132255) [ 1778.258853] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 0 sec [ 1778.800945] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 1784.363539] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 2 sec [ 1785.269331] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 0 sec [ 1825.784200] Lustre: DEBUG MARKER: ls -l 10000 files without statahead: 21 sec [ 1829.915095] Lustre: DEBUG MARKER: ls -l 10000 files with statahead: 3 sec [ 1830.249488] Lustre: DEBUG MARKER: ls -l done [ 1843.517867] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity/: 13 seconds [ 1844.051539] Lustre: DEBUG MARKER: rm done [ 1845.618131] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 11:45:23 (1737132323) [ 1845.976702] Lustre: DEBUG MARKER: SKIP: sanity test_123ab Test must be statx() syscall supported [ 1846.393570] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 11:45:24 (1737132324) [ 1846.753104] Lustre: DEBUG MARKER: SKIP: sanity test_123ac Test must be statx() syscall supported [ 1847.161230] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 11:45:25 (1737132325) [ 1850.660730] Lustre: DEBUG MARKER: ls done [ 1853.456398] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 11:45:31 (1737132331) [ 1853.890436] Lustre: DEBUG MARKER: SKIP: sanity test_123c needs >= 2 MDTs [ 1854.327535] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 11:45:32 (1737132332) [ 1854.720871] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 1860.783364] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8b28800 [ 1861.174509] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8b28800 [ 1861.575470] Lustre: DEBUG MARKER: LRU=2003 [ 1861.958550] Lustre: DEBUG MARKER: LIMIT=62166 [ 1862.348598] Lustre: DEBUG MARKER: LVF=3724300 [ 1862.733959] Lustre: DEBUG MARKER: OLD_LVF=100 [ 1863.116310] Lustre: DEBUG MARKER: Sleep 50 sec [ 1913.522830] Lustre: DEBUG MARKER: Dropped 802 locks in 50s [ 1913.847198] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 1918.204998] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 11:46:36 (1737132396) [ 1934.571194] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 1956.312686] Lustre: DEBUG MARKER: ls -la time: 21 seconds [ 1956.656682] Lustre: DEBUG MARKER: lru_size = 400 [ 1988.192802] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 1994.115370] Lustre: DEBUG MARKER: ls -la time: 5 seconds [ 1994.463344] Lustre: DEBUG MARKER: lru_size = 8005 [ 1994.799864] Lustre: DEBUG MARKER: ls -la is 76% faster with lru resize enabled [ 2005.565879] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 11:48:03 (1737132483) [ 2027.635091] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize diasbaled ========================================================== 11:48:25 (1737132505) [ 2049.730989] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 11:48:47 (1737132527) [ 2051.373619] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 11:48:49 (1737132529) [ 2052.943549] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 11:48:51 (1737132531) [ 2054.785509] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 11:48:53 (1737132533) [ 2056.314437] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [ 2199.715622] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 11:51:17 (1737132677) [ 2202.507781] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 11:51:20 (1737132680) [ 2203.138419] Lustre: DEBUG MARKER: SKIP: sanity test_129 ldiskfs only test [ 2203.961925] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 11:51:21 (1737132681) [ 2204.857419] Lustre: DEBUG MARKER: SKIP: sanity test_130a LU-1941: FIEMAP unimplemented on ZFS [ 2205.761462] Lustre: DEBUG MARKER: SKIP: sanity test_130b skipping ALWAYS excluded test 130b [ 2206.698884] Lustre: DEBUG MARKER: SKIP: sanity test_130c skipping ALWAYS excluded test 130c [ 2207.472804] Lustre: DEBUG MARKER: SKIP: sanity test_130d skipping ALWAYS excluded test 130d [ 2208.285934] Lustre: DEBUG MARKER: SKIP: sanity test_130e skipping ALWAYS excluded test 130e [ 2209.028826] Lustre: DEBUG MARKER: SKIP: sanity test_130f skipping ALWAYS excluded test 130f [ 2209.756027] Lustre: DEBUG MARKER: SKIP: sanity test_130g skipping ALWAYS excluded test 130g [ 2210.447050] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 11:51:28 (1737132688) [ 2213.242753] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 11:51:31 (1737132691) [ 2215.905415] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 11:51:33 (1737132693) [ 2218.283970] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 11:51:36 (1737132696) [ 2220.374974] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 11:51:38 (1737132698) [ 2222.440838] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 11:51:40 (1737132700) [ 2228.460443] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 11:51:46 (1737132706) [ 2237.315325] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 11:51:55 (1737132715) [ 2272.780994] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 11:52:30 (1737132750) [ 2283.512328] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 11:52:41 (1737132761) [ 2287.175007] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 11:52:45 (1737132765) [ 2293.250024] Lustre: server umount lustre-MDT0000 complete [ 2294.537835] LustreError: 6668:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1737132774 with bad export cookie 16117998882985526939 [ 2294.541668] LustreError: 166-1: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2300.537390] Lustre: 21063:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737132774/real 1737132774] req@ffff8801249ac740 x1821513376314816/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737132780 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2300.578631] Lustre: server umount lustre-OST0000 complete [ 2307.863419] Lustre: 21371:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737132781/real 1737132781] req@ffff8801175bb900 x1821513376315264/t0(0) o39->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737132787 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2307.964943] Lustre: server umount lustre-OST0001 complete [ 2311.169279] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2311.693558] Key type lgssc unregistered [ 2311.762752] LNet: 22011:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.765482] LNet: Removed LNI 192.168.201.138@tcp [ 2316.183592] alg: No test for adler32 (adler32-zlib) [ 2316.959460] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2317.093375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2317.202253] Lustre: Lustre: Build Version: 2.15.6_1_g67c9076 [ 2317.264494] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2317.266850] LNet: Accept secure, port 988 [ 2318.802389] Key type lgssc registered [ 2319.037520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2322.602013] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2326.027618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.209701] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2329.270326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.326019] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2331.263929] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2331.274834] Lustre: lustre-OST0000: deleting orphan objects from 0x0:26399 to 0x0:26433 [ 2332.533342] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2333.483130] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2336.350739] Lustre: lustre-OST0001: deleting orphan objects from 0x0:26436 to 0x0:26465 [ 2337.130104] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2338.045156] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2344.963158] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 11:53:43 (1737132823) [ 2356.866074] Lustre: server umount lustre-MDT0000 complete [ 2357.818811] LustreError: 23994:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1737132837 with bad export cookie 3264068138113598902 [ 2357.823805] LustreError: 166-1: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2363.818371] Lustre: 29626:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737132837/real 1737132837] req@ffff8801250d9c80 x1821515758713792/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737132843 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2363.832230] Lustre: server umount lustre-OST0000 complete [ 2370.767378] Lustre: 29933:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737132844/real 1737132844] req@ffff88006de83dc0 x1821515758714176/t0(0) o39->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737132850 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2370.813557] Lustre: server umount lustre-OST0001 complete [ 2373.653580] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2374.053397] Key type lgssc unregistered [ 2374.101505] LNet: 30570:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.103687] LNet: Removed LNI 192.168.201.138@tcp [ 2378.149920] alg: No test for adler32 (adler32-zlib) [ 2378.922381] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2379.079035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2379.130465] Lustre: Lustre: Build Version: 2.15.6_1_g67c9076 [ 2379.169001] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2379.171000] LNet: Accept secure, port 988 [ 2380.694364] Key type lgssc registered [ 2380.865500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2383.846115] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2386.839623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2387.791737] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2389.589315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2390.496299] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2391.582329] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.589639] Lustre: lustre-OST0000: deleting orphan objects from 0x0:26399 to 0x0:26465 [ 2392.220658] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2393.092247] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2394.904053] Lustre: lustre-OST0001: deleting orphan objects from 0x0:26436 to 0x0:26497 [ 2396.620580] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2397.487782] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2404.089807] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 11:54:42 (1737132882) [ 2560.621709] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 11:57:18 (1737133038) [ 2563.826411] Lustre: *** cfs_fail_loc=327, val=0*** [ 2578.091069] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 11:57:36 (1737133056) [ 2579.630984] Lustre: *** cfs_fail_loc=328, val=0*** [ 2579.632937] Lustre: Skipped 512 previous similar messages [ 2581.544374] Lustre: *** cfs_fail_loc=328, val=0*** [ 2581.546379] Lustre: Skipped 501 previous similar messages [ 2583.555245] Lustre: *** cfs_fail_loc=328, val=0*** [ 2583.557025] Lustre: Skipped 1 previous similar message [ 2587.572066] Lustre: *** cfs_fail_loc=328, val=0*** [ 2587.573511] Lustre: Skipped 3 previous similar messages [ 2595.602371] Lustre: *** cfs_fail_loc=328, val=0*** [ 2595.604112] Lustre: Skipped 7 previous similar messages [ 2599.751172] LustreError: 16768:0:(ldlm_resource.c:133:seq_watermark_write()) Failed to set lock_reclaim_threshold_mb, rc = -22. [ 2603.062727] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [ 2603.459958] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [ 2603.866898] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 11:58:02 (1737133082) [ 2608.807255] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 11:58:07 (1737133087) [ 2628.326117] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 11:58:26 (1737133106) [ 2628.734590] Lustre: DEBUG MARKER: SKIP: sanity test_150b need >= 2.13.57 and ldiskfs for fallocate [ 2629.181897] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 11:58:27 (1737133107) [ 2629.574560] Lustre: DEBUG MARKER: SKIP: sanity test_150bb need >= 2.13.57 and ldiskfs for fallocate [ 2630.031356] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 11:58:28 (1737133108) [ 2630.436837] Lustre: DEBUG MARKER: SKIP: sanity test_150c need >= 2.13.57 and ldiskfs for fallocate [ 2630.919548] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 11:58:29 (1737133109) [ 2631.351788] Lustre: DEBUG MARKER: SKIP: sanity test_150d need >= 2.13.57 and ldiskfs for fallocate [ 2631.810326] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 11:58:30 (1737133110) [ 2632.205652] Lustre: DEBUG MARKER: SKIP: sanity test_150e need >= 2.13.57 and ldiskfs for fallocate [ 2632.617531] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 11:58:30 (1737133110) [ 2632.966392] Lustre: DEBUG MARKER: SKIP: sanity test_150f LU-14160: punch mode is not implemented on OSD ZFS [ 2633.400767] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 11:58:31 (1737133111) [ 2633.763173] Lustre: DEBUG MARKER: SKIP: sanity test_150g LU-14160: punch mode is not implemented on OSD ZFS [ 2634.213898] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 11:58:32 (1737133112) [ 2634.982177] Lustre: DEBUG MARKER: SKIP: sanity test_151 not cache-capable obdfilter [ 2635.434246] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 11:58:33 (1737133113) [ 2637.043534] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 11:58:35 (1737133115) [ 2638.634234] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 11:58:36 (1737133116) [ 2640.094776] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 11:58:38 (1737133118) [ 2641.592857] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 11:58:39 (1737133119) [ 2641.776508] LustreError: 1166:0:(fld_handler.c:263:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 2643.519631] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 11:58:41 (1737133121) [ 2643.861559] Lustre: DEBUG MARKER: SKIP: sanity test_154b needs >= 2 MDTs [ 2644.333455] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 11:58:42 (1737133122) [ 2646.089488] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 11:58:44 (1737133124) [ 2647.981428] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 11:58:46 (1737133126) [ 2650.035415] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 11:58:48 (1737133128) [ 2653.180962] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 11:58:51 (1737133131) [ 2781.054417] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 12:00:58 (1737133258) [ 2789.921548] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 12:01:07 (1737133267) [ 2796.957186] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 12:01:14 (1737133274) [ 2802.487165] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 12:01:20 (1737133280) [ 2808.217040] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 12:01:26 (1737133286) [ 2828.598641] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 12:01:46 (1737133306) [ 2852.344613] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 12:02:10 (1737133330) [ 2875.854243] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 12:02:34 (1737133354) [ 2896.244162] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 12:02:54 (1737133374) [ 2896.756253] Lustre: DEBUG MARKER: SKIP: sanity test_156 LU-1956/LU-2261: stats not implemented on OSD ZFS [ 2897.290172] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 12:02:55 (1737133375) [ 2898.167654] Lustre: lustre-MDD0000: changelog on [ 2902.391843] Lustre: Failing over lustre-MDT0000 [ 2902.509529] Lustre: server umount lustre-MDT0000 complete [ 2904.542028] LustreError: 166-1: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2904.549073] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x89262a9e1f78dc79 to 0x89262a9e1f816281 [ 2904.557663] Lustre: MGC192.168.201.138@tcp: Connection restored to 192.168.201.138@tcp (at 0@lo) [ 2904.583767] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.711476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2904.740431] Lustre: lustre-MDD0000: changelog on [ 2904.744362] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2905.909221] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2906.100999] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2906.177481] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2906.201666] Lustre: lustre-OST0001: deleting orphan objects from 0x0:27318 to 0x0:27361 [ 2906.201713] Lustre: lustre-OST0000: deleting orphan objects from 0x0:27282 to 0x0:27297 [ 2907.651635] Lustre: lustre-MDD0000: changelog off [ 2909.689945] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.695367] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2909.698794] Lustre: Skipped 1 previous similar message [ 2911.522049] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 12:03:09 (1737133389) [ 2912.639523] Lustre: lustre-MDD0000: changelog on [ 2916.507135] Lustre: lustre-MDD0000: changelog off [ 2917.586839] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 12:03:15 (1737133395) [ 2918.829017] Lustre: lustre-MDD0000: changelog on [ 2924.458163] Lustre: lustre-MDD0000: changelog off [ 2925.773620] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 12:03:23 (1737133403) [ 2926.694235] Lustre: DEBUG MARKER: SKIP: sanity test_160d needs >= 2 MDTs [ 2927.586279] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 12:03:25 (1737133405) [ 2928.980888] Lustre: lustre-MDD0000: changelog on [ 2931.905967] Lustre: lustre-MDD0000: changelog off [ 2932.865878] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 12:03:30 (1737133410) [ 2935.891599] Lustre: DEBUG MARKER: 1737133413: creating first files [ 2946.739675] Lustre: *** cfs_fail_loc=1313, val=0*** [ 2946.741858] Lustre: 612:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 2946.743497] Lustre: 31098:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737133382/real 1737133382] req@ffff8800a599e880 x1821515823912576/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737133426 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 2946.758734] Lustre: 6114:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl6 idle for 12s with 4 unprocessed records [ 2948.748525] Lustre: 31100:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737133384/real 1737133384] req@ffff880086be9300 x1821515823912768/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737133428 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 2948.764003] Lustre: 31100:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2952.227888] Lustre: lustre-MDD0000: changelog off [ 2953.103101] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 12:03:51 (1737133431) [ 2954.202511] Lustre: lustre-MDD0000: changelog on [ 2954.204039] Lustre: Skipped 1 previous similar message [ 2959.656801] Lustre: 612:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 2959.662349] Lustre: 8266:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl8 idle for 5s with 4 unprocessed records [ 2965.074732] Lustre: lustre-MDD0000: changelog off [ 2965.927283] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 12:04:04 (1737133444) [ 2982.689794] Lustre: *** cfs_fail_loc=1316, val=0*** [ 2982.691569] Lustre: 611:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: simulate starting changelog garbage collection [ 2982.695761] Lustre: 10470:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl10 idle for 15s with 4 unprocessed records [ 2983.264476] Lustre: Failing over lustre-MDT0000 [ 2984.716641] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) [ 2984.810225] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2984.815073] Lustre: Skipped 1 previous similar message [ 2986.263666] Lustre: server umount lustre-MDT0000 complete [ 2988.839657] LustreError: 166-1: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2988.844607] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x89262a9e1f816281 to 0x89262a9e1f816db7 [ 2988.848633] Lustre: MGC192.168.201.138@tcp: Connection restored to (at 0@lo) [ 2988.850874] Lustre: Skipped 1 previous similar message [ 2988.993797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2989.022298] Lustre: lustre-MDD0000: changelog on [ 2989.023930] Lustre: Skipped 1 previous similar message [ 2989.026350] Lustre: 11918:0:(mdd_device.c:618:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 34 to index 35, being cleared now [ 2989.035433] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2989.716798] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2989.780331] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2989.796576] Lustre: lustre-OST0001: deleting orphan objects from 0x0:27318 to 0x0:27393 [ 2989.797212] Lustre: lustre-OST0000: deleting orphan objects from 0x0:27299 to 0x0:27329 [ 2990.085555] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2994.832701] Lustre: lustre-MDD0000: changelog off [ 2995.528535] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 12:04:33 (1737133473) [ 2997.433011] LustreError: 14021:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 1315 sleeping [ 2999.665601] LustreError: 14190:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 1315 waking [ 2999.668133] LustreError: 14021:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 1315 awake: rc=2768 [ 2999.984554] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 3003.628503] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 12:04:41 (1737133481) [ 3008.147592] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 12:04:46 (1737133486) [ 3009.625522] LustreError: 11928:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 15d sleeping for 3000ms [ 3012.629384] LustreError: 11928:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 15d awake [ 3018.974084] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 12:04:57 (1737133497) [ 3024.741821] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 12:05:02 (1737133502) [ 3025.523479] Lustre: lustre-MDD0000: changelog on [ 3025.524785] Lustre: Skipped 5 previous similar messages [ 3026.989229] LustreError: 11926:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 15f sleeping [ 3028.991403] LustreError: 12207:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15f waking [ 3028.994072] LustreError: 11926:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 15f awake: rc=2998 [ 3031.588479] Lustre: lustre-MDD0000: changelog off [ 3031.589868] Lustre: Skipped 5 previous similar messages [ 3032.291425] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 12:05:10 (1737133510) [ 3318.212704] LustreError: 20926:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 16c sleeping [ 3320.213955] LustreError: 11926:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 16c waking [ 3320.216124] LustreError: 20926:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 16c awake: rc=2999 [ 3321.797383] LustreError: 20926:0:(osd_handler.c:228:osd_trans_start()) lustre-MDT0000: can't assign tx: rc = -2 [ 3323.877584] Lustre: lustre-MDD0000: changelog off [ 3324.552907] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 12:10:02 (1737133802) [ 3325.316857] Lustre: lustre-MDD0000: changelog on [ 3325.318334] Lustre: Skipped 1 previous similar message [ 3325.557359] LustreError: 23567:0:(mdd_device.c:1704:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [ 3325.790138] Lustre: 23631:0:(mdd_device.c:1721:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [ 3326.027092] LustreError: 23696:0:(mdd_device.c:1713:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [ 3331.593398] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 12:10:09 (1737133809) [ 3332.043615] Lustre: DEBUG MARKER: SKIP: sanity test_160p ldiskfs only test [ 3332.520931] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 12:10:10 (1737133810) [ 3335.334590] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records anaconda-ks.cfg stress.sh time ========================================================== 12:10:13 (1737133813) [ 3341.001764] Lustre: 12207:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 3341.006099] Lustre: 26515:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl24 idle for 864004s with 500000004 unprocessed records [ 3345.062470] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 12:10:23 (1737133823) [ 3352.350152] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 12:10:30 (1737133830) [ 3352.741060] Lustre: DEBUG MARKER: SKIP: sanity test_161b skipping remote directory test [ 3353.196613] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 12:10:31 (1737133831) [ 3356.721443] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 12:10:34 (1737133834) [ 3361.913059] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 12:10:40 (1737133840) [ 3363.743074] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 12:10:41 (1737133841) [ 3364.112811] Lustre: DEBUG MARKER: SKIP: sanity test_162b needs >= 2 MDTs [ 3364.539138] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 12:10:42 (1737133842) [ 3370.735882] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 12:10:48 (1737133848) [ 3377.065198] Lustre: Failing over lustre-OST0000 [ 3377.076984] Lustre: server umount lustre-OST0000 complete [ 3379.575672] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3379.579893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3379.601363] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3381.976014] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3381.989368] LustreError: Skipped 1 previous similar message [ 3386.975211] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.025532] LustreError: Skipped 1 previous similar message [ 3387.809490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3387.832176] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3389.281442] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3389.881452] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3389.882340] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.138@tcp (at 0@lo) [ 3389.882341] Lustre: Skipped 1 previous similar message [ 3389.939680] Lustre: lustre-OST0000: deleting orphan objects from 0x0:27337 to 0x0:27361 [ 3390.549107] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3393.432453] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 12:11:10 (1737133870) [ 3419.645360] Lustre: Failing over lustre-OST0000 [ 3419.675404] Lustre: server umount lustre-OST0000 complete [ 3419.961750] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3419.965207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.974541] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3422.642745] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 3422.652305] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3424.007443] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3424.107901] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3424.663484] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3424.663915] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.138@tcp (at 0@lo) [ 3424.675839] Lustre: lustre-OST0000: deleting orphan objects from 0x0:27363 to 0x0:27393 [ 3425.352660] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 12:11:43 (1737133903) [ 3437.016383] Lustre: Failing over lustre-OST0000 [ 3437.043364] Lustre: server umount lustre-OST0000 complete [ 3437.075357] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3437.082814] LustreError: Skipped 1 previous similar message [ 3437.657351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.315059] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 3440.321486] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3441.769737] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3441.886337] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3442.125961] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3442.130209] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.138@tcp (at 0@lo) [ 3442.146879] Lustre: lustre-OST0000: deleting orphan objects from 0x0:27522 to 0x0:27553 [ 3443.926986] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 12:12:01 (1737133921) [ 3470.569225] Lustre: Failing over lustre-OST0000 [ 3470.599617] Lustre: server umount lustre-OST0000 complete [ 3472.110996] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3472.120869] LustreError: Skipped 1 previous similar message [ 3472.184901] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3472.189362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.442246] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 3473.447844] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3474.702191] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3474.795665] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3475.201784] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3475.201902] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.138@tcp (at 0@lo) [ 3475.205707] Lustre: lustre-OST0000: deleting orphan objects from 0x0:27555 to 0x0:27585 [ 3476.264375] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 12:12:34 (1737133954) [ 3476.867117] Lustre: DEBUG MARKER: SKIP: sanity test_165e needs >= 2 MDTs [ 3477.676392] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 12:12:35 (1737133955) [ 3485.223644] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3488.637914] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 3489.605098] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3490.283403] Lustre: lustre-OST0000: deleting orphan objects from 0x0:27555 to 0x0:27617 [ 3490.656095] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 12:12:48 (1737133968) [ 3491.137314] Lustre: DEBUG MARKER: creating a 10 Mb file [ 3491.714473] Lustre: DEBUG MARKER: starting reads [ 3492.255597] Lustre: DEBUG MARKER: truncating the file [ 3492.771723] Lustre: DEBUG MARKER: killing dd [ 3493.377235] Lustre: DEBUG MARKER: removing the temporary file [ 3496.140520] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log =============================================================================== 12:12:54 (1737133974) [ 3499.153514] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 12:12:57 (1737133977) [ 3504.915101] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 12:13:02 (1737133982) [ 3505.441254] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [ 3506.019346] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 12:13:04 (1737133984) [ 3507.018131] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_module obdecho/obdecho [ 3513.352744] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 12:13:11 (1737133991) [ 3514.313649] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_module obdecho/obdecho [ 3514.352822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3521.947386] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 12:13:19 (1737133999) [ 3572.461188] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ========================================================================== 12:14:10 (1737134050) [ 3604.455690] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 12:14:42 (1737134082) [ 3605.080988] Lustre: *** cfs_fail_loc=148, val=0*** [ 3608.479139] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 12:14:46 (1737134086) [ 3612.540125] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 12:14:50 (1737134090) [ 3615.031761] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 12:14:53 (1737134093) [ 3625.498862] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 12:15:03 (1737134103) [ 3636.245590] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 12:15:13 (1737134113) [ 3646.523407] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 12:15:24 (1737134124) [ 3649.259464] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 12:15:27 (1737134127) [ 3653.144831] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 12:15:31 (1737134131) [ 3658.389758] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 12:15:36 (1737134136) [ 3661.532655] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 12:15:39 (1737134139) [ 3664.277484] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 12:15:42 (1737134142) [ 3682.149769] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 12:16:00 (1737134160) [ 3685.339137] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 12:16:03 (1737134163) [ 3688.300748] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 12:16:06 (1737134166) [ 3691.144805] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 12:16:09 (1737134169) [ 3694.152712] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 12:16:12 (1737134172) [ 3696.907094] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 12:16:14 (1737134174) [ 3699.434475] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 12:16:17 (1737134177) [ 3701.750438] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 12:16:19 (1737134179) [ 3703.947909] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 12:16:22 (1737134182) [ 3707.567399] Lustre: lustre-MDD0000: changelog on [ 3707.569363] Lustre: Skipped 4 previous similar messages [ 3708.921971] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [ 3709.639502] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.8568 [ 3710.754746] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [ 3711.386479] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.3137 [ 3712.457094] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [ 3713.069648] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.710 [ 3713.900893] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [ 3714.373505] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.9624 [ 3715.152627] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [ 3715.643671] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.18278 [ 3716.507072] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3717.101983] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.31942 [ 3718.680891] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [ 3719.374096] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.23612 [ 3720.658228] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [ 3721.471632] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.29079 [ 3722.631755] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [ 3723.354198] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.4735 [ 3724.766227] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [ 3725.337252] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.337 [ 3726.457392] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [ 3727.051255] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.28117 [ 3730.991544] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3731.635696] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg138-client.v [ 3732.709010] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3733.314068] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg138-client.E [ 3734.168991] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3734.782616] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.oleg138-client.v [ 3737.699006] Lustre: lustre-MDD0000: changelog off [ 3737.701334] Lustre: Skipped 5 previous similar messages [ 3738.873104] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 12:16:56 (1737134216) [ 3743.402196] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 12:17:01 (1737134221) [ 3745.659812] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 12:17:03 (1737134223) [ 3747.983782] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 12:17:06 (1737134226) [ 3749.928497] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 12:17:08 (1737134228) [ 3751.935576] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 12:17:10 (1737134230) [ 3758.787031] Lustre: Failing over lustre-MDT0000 [ 3758.788453] Lustre: Skipped 1 previous similar message [ 3758.985574] Lustre: server umount lustre-MDT0000 complete [ 3758.987173] Lustre: Skipped 1 previous similar message [ 3766.063573] Lustre: 31100:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737134238/real 1737134238] req@ffff88006411a140 x1821515824974720/t0(0) o400->MGC192.168.201.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1737134245 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 3766.073983] Lustre: 31100:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3766.076792] LustreError: 166-1: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3770.081396] Lustre: 31100:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737134238/real 1737134238] req@ffff880064118e40 x1821515824974784/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737134249 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 3770.082372] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3770.082374] Lustre: Skipped 1 previous similar message [ 3770.108712] Lustre: 31100:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3775.082495] Lustre: 31101:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737134243/real 1737134243] req@ffff88006411f6c0 x1821515824974976/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737134254 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 3777.082857] Lustre: 31099:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737134245/real 1737134245] req@ffff88012c96df00 x1821515824975168/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737134256 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 3777.094308] Lustre: 31099:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3781.093845] Lustre: 31101:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737134249/real 1737134249] req@ffff88006411b900 x1821515824975296/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737134260 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 3783.114592] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x89262a9e1f816db7 to 0x89262a9e1fd7a4cc [ 3783.120519] Lustre: MGC192.168.201.138@tcp: Connection restored to 192.168.201.138@tcp (at 0@lo) [ 3783.123794] Lustre: Skipped 1 previous similar message [ 3783.273194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3783.314585] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3783.317655] Lustre: Skipped 1 previous similar message [ 3784.500322] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3790.630622] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3790.637803] Lustre: Skipped 1 previous similar message [ 3790.712756] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3790.716465] Lustre: Skipped 1 previous similar message [ 3790.738221] Lustre: lustre-OST0001: deleting orphan objects from 0x0:34556 to 0x0:34593 [ 3790.738681] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34513 to 0x0:34529 [ 3792.425200] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3792.925104] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3796.848736] Lustre: Failing over lustre-MDT0000 [ 3796.997388] Lustre: server umount lustre-MDT0000 complete [ 3804.279455] Lustre: 31098:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737134276/real 1737134276] req@ffff88012c96c740 x1821515824980928/t0(0) o400->MGC192.168.201.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1737134283 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 3804.289745] LustreError: 166-1: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3808.280288] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3808.286369] Lustre: Skipped 1 previous similar message [ 3810.302119] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x89262a9e1fd7a4cc to 0x89262a9e1fd7a88b [ 3811.818072] Lustre: lustre-OST0001: deleting orphan objects from 0x0:34556 to 0x0:34625 [ 3811.821349] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34513 to 0x0:34561 [ 3812.316333] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3816.345746] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3816.899757] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3820.814384] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 12:18:18 (1737134298) [ 3832.030065] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 12:18:30 (1737134310) [ 3836.136116] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 12:18:34 (1737134314) [ 3839.108194] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 12:18:37 (1737134317) [ 3852.260196] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 12:18:50 (1737134330) [ 3862.706803] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 12:19:00 (1737134340) [ 3865.092997] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 12:19:03 (1737134343) [ 3872.903466] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') ========================================================== 12:19:10 (1737134350) [ 3876.105832] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 12:19:14 (1737134354) [ 3876.789169] Lustre: DEBUG MARKER: creating a 10 Mb file [ 3877.653385] Lustre: DEBUG MARKER: starting reads [ 3878.376523] Lustre: DEBUG MARKER: truncating the file [ 3879.113182] Lustre: DEBUG MARKER: killing dd [ 3879.742022] Lustre: DEBUG MARKER: removing the temporary file [ 3882.334132] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 12:19:20 (1737134360) [ 3884.813290] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 12:19:22 (1737134362) [ 3890.437470] Lustre: *** cfs_fail_loc=229, val=0*** [ 3900.171974] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 12:19:38 (1737134378) [ 3907.104120] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 12:19:45 (1737134385) [ 3910.249441] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 12:19:48 (1737134388) [ 3913.385161] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 12:19:51 (1737134391) [ 3916.499054] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 12:19:54 (1737134394) [ 3916.605190] Lustre: lustre-OST0001: Client 77cf3c50-764f-48a2-b0d6-8f1d2647fdaf (at 192.168.201.38@tcp) reconnecting [ 3920.087647] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 12:19:58 (1737134398) [ 3921.933112] LustreError: 1146:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 sleeping for 3000ms [ 3924.940161] LustreError: 1146:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 awake [ 3928.322026] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 12:20:06 (1737134406) [ 3934.244330] Lustre: *** cfs_fail_loc=520, val=57344*** [ 3934.248083] LNet: *** cfs_fail_loc=e000, val=2147483648*** [ 3939.246301] Lustre: lustre-MDT0000: Client 77cf3c50-764f-48a2-b0d6-8f1d2647fdaf (at 192.168.201.38@tcp) reconnecting [ 3939.280860] Lustre: 3271:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008882f6c0 x1821515846737856/t30064772699(0) o35->77cf3c50-764f-48a2-b0d6-8f1d2647fdaf@192.168.201.38@tcp:0/0 lens 392/456 e 0 to 0 dl 1737134423 ref 1 fl Interpret:/2/0 rc 0/0 job:'' [ 3947.558105] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 12:20:25 (1737134425) [ 3949.480508] LustreError: 27081:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 sleeping for 22000ms [ 3970.302635] Lustre: lustre-OST0000: Client 77cf3c50-764f-48a2-b0d6-8f1d2647fdaf (at 192.168.201.38@tcp) reconnecting [ 3971.504373] LustreError: 27081:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 awake [ 3971.507817] LustreError: 27081:0:(ldlm_lib.c:3493:target_bulk_io()) @@@ bulk READ failed: rc = -107 req@ffff880124343440 x1821515846742848/t0(0) o3->77cf3c50-764f-48a2-b0d6-8f1d2647fdaf@192.168.201.38@tcp:0/0 lens 488/440 e 0 to 0 dl 1737134449 ref 1 fl Interpret:/0/0 rc 0/0 job:'' [ 3971.523363] Lustre: 27081:0:(service.c:2333:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/2s); client may timeout req@ffff880124343440 x1821515846742848/t0(0) o3->77cf3c50-764f-48a2-b0d6-8f1d2647fdaf@192.168.201.38@tcp:0/0 lens 488/440 e 0 to 0 dl 1737134449 ref 1 fl Complete:/0/0 rc 0/0 job:'' [ 3974.893357] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [ 3975.521877] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [ 3976.095011] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 12:20:54 (1737134454) [ 3978.594835] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 12:20:56 (1737134456) [ 3979.115560] Lustre: DEBUG MARKER: SKIP: sanity test_226b needs >= 2 MDTs [ 3979.720122] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 12:20:57 (1737134457) [ 3980.258473] Lustre: DEBUG MARKER: SKIP: sanity test_226c needs >= 2 MDTs [ 3980.847225] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 12:20:58 (1737134458) [ 3983.028478] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 12:21:01 (1737134461) [ 3983.640354] Lustre: DEBUG MARKER: SKIP: sanity test_228a ldiskfs only test [ 3984.232246] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 12:21:02 (1737134462) [ 3984.825956] Lustre: DEBUG MARKER: SKIP: sanity test_228b ldiskfs only test [ 3985.500696] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 12:21:03 (1737134463) [ 3986.170915] Lustre: DEBUG MARKER: SKIP: sanity test_228c ldiskfs only test [ 3986.833954] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 12:21:04 (1737134464) [ 3989.179023] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 12:21:07 (1737134467) [ 3989.770846] Lustre: DEBUG MARKER: SKIP: sanity test_230a needs >= 2 MDTs [ 3990.471655] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 12:21:08 (1737134468) [ 3990.972510] Lustre: DEBUG MARKER: SKIP: sanity test_230b needs >= 2 MDTs [ 3991.541689] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 12:21:09 (1737134469) [ 3992.037786] Lustre: DEBUG MARKER: SKIP: sanity test_230c needs >= 2 MDTs [ 3992.660119] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [ 3993.246870] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 12:21:11 (1737134471) [ 3993.795990] Lustre: DEBUG MARKER: SKIP: sanity test_230e needs >= 2 MDTs [ 3994.437891] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 12:21:12 (1737134472) [ 3994.929094] Lustre: DEBUG MARKER: SKIP: sanity test_230f needs >= 2 MDTs [ 3995.515974] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 12:21:13 (1737134473) [ 3996.008832] Lustre: DEBUG MARKER: SKIP: sanity test_230g needs >= 2 MDTs [ 3996.634147] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 12:21:14 (1737134474) [ 3997.216690] Lustre: DEBUG MARKER: SKIP: sanity test_230h needs >= 2 MDTs [ 3997.847836] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 12:21:15 (1737134475) [ 3998.426754] Lustre: DEBUG MARKER: SKIP: sanity test_230i needs >= 2 MDTs [ 3999.141038] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 12:21:17 (1737134477) [ 3999.747017] Lustre: DEBUG MARKER: SKIP: sanity test_230j needs >= 2 MDTs [ 4000.407071] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 12:21:18 (1737134478) [ 4000.965976] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [ 4001.634687] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 12:21:19 (1737134479) [ 4002.170350] Lustre: DEBUG MARKER: SKIP: sanity test_230l needs >= 2 MDTs [ 4002.785967] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 12:21:20 (1737134480) [ 4003.302466] Lustre: DEBUG MARKER: SKIP: sanity test_230m needs >= 2 MDTs [ 4003.907198] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 12:21:22 (1737134482) [ 4004.401927] Lustre: DEBUG MARKER: SKIP: sanity test_230n needs >= 2 MDTs [ 4005.001559] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 12:21:23 (1737134483) [ 4005.485396] Lustre: DEBUG MARKER: SKIP: sanity test_230o needs >= 2 MDTs [ 4006.084187] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 12:21:24 (1737134484) [ 4006.594108] Lustre: DEBUG MARKER: SKIP: sanity test_230p needs >= 2 MDTs [ 4007.197476] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 12:21:25 (1737134485) [ 4007.669298] Lustre: DEBUG MARKER: SKIP: sanity test_230q needs >= 2 MDTs [ 4008.219953] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 12:21:26 (1737134486) [ 4008.704383] Lustre: DEBUG MARKER: SKIP: sanity test_230r needs >= 2 MDTs [ 4009.267700] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 12:21:27 (1737134487) [ 4013.240536] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 12:21:31 (1737134491) [ 4013.839675] Lustre: DEBUG MARKER: SKIP: sanity test_230t needs >= 2 MDTs [ 4014.593188] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 12:21:32 (1737134492) [ 4015.231312] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [ 4015.937837] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 12:21:33 (1737134493) [ 4016.623765] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [ 4017.293041] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 12:21:35 (1737134495) [ 4017.913921] Lustre: DEBUG MARKER: SKIP: sanity test_230w needs >= 2 MDTs [ 4018.775795] Lustre: DEBUG MARKER: == sanity test 230y: unlink dir with bad hash type ======= 12:21:36 (1737134496) [ 4019.489465] Lustre: DEBUG MARKER: SKIP: sanity test_230y needs >= 2 MDTs [ 4020.164049] Lustre: DEBUG MARKER: == sanity test 230z: resume dir migration with bad hash type ========================================================== 12:21:38 (1737134498) [ 4020.683281] Lustre: DEBUG MARKER: SKIP: sanity test_230z needs >= 2 MDTs [ 4021.325593] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 12:21:39 (1737134499) [ 4025.235592] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 12:21:43 (1737134503) [ 4033.194853] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 12:21:51 (1737134511) [ 4033.619384] Lustre: *** cfs_fail_loc=31c, val=0*** [ 4035.299598] Lustre: Failing over lustre-OST0000 [ 4035.357753] Lustre: server umount lustre-OST0000 complete [ 4035.863898] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4035.867804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4035.876389] Lustre: Skipped 1 previous similar message [ 4035.879349] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.885569] LustreError: Skipped 4 previous similar messages [ 4038.473606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.476182] Lustre: Skipped 1 previous similar message [ 4038.480682] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4038.484158] Lustre: Skipped 1 previous similar message [ 4039.515353] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4039.519357] Lustre: Skipped 1 previous similar message [ 4039.598018] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4039.599574] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.138@tcp (at 0@lo) [ 4039.599576] Lustre: Skipped 5 previous similar messages [ 4039.606704] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34790 to 0x0:34817 [ 4039.611210] Lustre: Skipped 1 previous similar message [ 4040.002506] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4043.322558] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 12:22:01 (1737134521) [ 4043.803559] Lustre: *** cfs_fail_loc=31c, val=0*** [ 4043.806970] LustreError: 1827:0:(ldlm_request.c:490:ldlm_cli_enqueue_local()) ### delayed lvb init failed (rc -12) ns: filter-lustre-OST0000_UUID lock: ffff880134c58480/0x89262a9e1fd83c79 lrc: 2/0,0 mode: --/PR res: [0x8802:0x0:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 1827 timeout: 0 lvb_type: 0 [ 4049.526243] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4049.990632] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34819 to 0x0:34849 [ 4052.959672] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 12:22:10 (1737134530) [ 4055.656854] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 12:22:13 (1737134533) [ 4057.975685] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 12:22:16 (1737134536) [ 4060.733237] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 12:22:18 (1737134538) [ 4065.116530] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 12:22:23 (1737134543) [ 4067.697102] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 12:22:25 (1737134545) [ 4070.148089] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 12:22:28 (1737134548) [ 4113.436875] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 12:23:11 (1737134591) [ 4114.693773] Lustre: *** cfs_fail_loc=2100, val=0*** [ 4130.381213] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 12:23:28 (1737134608) [ 4148.233181] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 12:23:46 (1737134626) [ 4148.706626] Lustre: DEBUG MARKER: SKIP: sanity test_240 needs >= 2 MDTs [ 4149.278690] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 12:23:47 (1737134627) [ 4179.906581] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 12:24:17 (1737134657) [ 4195.888448] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 12:24:33 (1737134673) [ 4196.446355] Lustre: *** cfs_fail_loc=105, val=0*** [ 4199.822807] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 12:24:37 (1737134677) [ 4240.146343] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 12:25:18 (1737134718) [ 4254.517420] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 12:25:32 (1737134732) [ 4257.511818] Lustre: DEBUG MARKER: == sanity test 245: check mdc connection flag/data: multiple modify RPCs ========================================================== 12:25:35 (1737134735) [ 4259.993186] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 12:25:38 (1737134738) [ 4263.033777] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 12:25:41 (1737134741) [ 4265.350235] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 12:25:43 (1737134743) [ 4267.828698] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 12:25:45 (1737134745) [ 4270.693149] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 12:25:48 (1737134748) [ 4273.411731] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 12:25:51 (1737134751) [ 4273.959426] Lustre: DEBUG MARKER: SKIP: sanity test_247f needs >= 2 MDTs [ 4274.569303] Lustre: DEBUG MARKER: == sanity test 247g: mount striped directory as fileset caches ROOT lookup lock ========================================================== 12:25:52 (1737134752) [ 4275.102344] Lustre: DEBUG MARKER: SKIP: sanity test_247g needs >= 4 MDTs [ 4275.781312] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 12:25:53 (1737134753) [ 4285.366752] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 12:26:03 (1737134763) [ 4319.646006] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 12:26:37 (1737134797) [ 4321.789122] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 12:26:39 (1737134799) [ 4322.345822] Lustre: DEBUG MARKER: SKIP: sanity test_250 no 16TB file size limit on ZFS [ 4323.023220] Lustre: DEBUG MARKER: == sanity test 251: Handling short read and write correctly ========================================================== 12:26:41 (1737134801) [ 4325.271657] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 12:26:43 (1737134803) [ 4325.931081] Lustre: DEBUG MARKER: SKIP: sanity test_252 ldiskfs only test [ 4326.539746] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 12:26:44 (1737134804) [ 4384.230396] LustreError: 3268:0:(lod_qos.c:1362:lod_ost_alloc_specific()) can't lstripe objid [0x2000032e2:0x13c2:0x0]: have 0 want 1 [ 4448.790850] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 12:28:46 (1737134926) [ 4449.946538] Lustre: lustre-MDD0000: changelog on [ 4452.977779] Lustre: lustre-MDD0000: changelog off [ 4453.920091] Lustre: DEBUG MARKER: SKIP: sanity test_255a skipping excluded test 255a (base 255) [ 4454.551718] Lustre: DEBUG MARKER: SKIP: sanity test_255b skipping excluded test 255b (base 255) [ 4455.194106] Lustre: DEBUG MARKER: SKIP: sanity test_255c skipping excluded test 255c (base 255) [ 4455.877073] Lustre: DEBUG MARKER: SKIP: sanity test_256 skipping excluded test 256 [ 4456.455845] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 12:28:54 (1737134934) [ 4456.974543] Lustre: *** cfs_fail_loc=161, val=0*** [ 4457.573390] Lustre: Failing over lustre-MDT0000 [ 4457.575188] Lustre: Skipped 1 previous similar message [ 4457.677005] mdt00_003 (3339) used greatest stack depth: 9296 bytes left [ 4457.722998] Lustre: server umount lustre-MDT0000 complete [ 4457.724837] Lustre: Skipped 1 previous similar message [ 4460.168649] LustreError: 166-1: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4460.174305] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x89262a9e1fd7a88b to 0x89262a9e1fdf89db [ 4460.178415] Lustre: MGC192.168.201.138@tcp: Connection restored to 192.168.201.138@tcp (at 0@lo) [ 4460.181243] Lustre: Skipped 1 previous similar message [ 4460.306245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4460.309164] Lustre: Skipped 1 previous similar message [ 4460.354726] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4460.357414] Lustre: Skipped 1 previous similar message [ 4461.376106] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4461.379238] Lustre: Skipped 1 previous similar message [ 4461.431928] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4461.447256] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4461.450661] Lustre: Skipped 1 previous similar message [ 4461.464584] Lustre: lustre-OST0001: deleting orphan objects from 0x0:37323 to 0x0:37345 [ 4461.464983] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37375 to 0x0:37409 [ 4463.191447] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 4465.287837] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4465.295356] Lustre: Skipped 2 previous similar messages [ 4465.367524] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 12:29:03 (1737134943) [ 4466.172501] Lustre: 31099:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737134938/real 1737134938] req@ffff8800850384c0 x1821515825686144/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737134945 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 4466.182984] Lustre: 31099:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 4467.746707] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 12:29:05 (1737134945) [ 4470.250890] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 12:29:08 (1737134948) [ 4470.885645] Lustre: DEBUG MARKER: SKIP: sanity test_259 ldiskfs only test [ 4471.504143] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 12:29:09 (1737134949) [ 4474.038567] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 12:29:12 (1737134952) [ 4477.757948] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 12:29:15 (1737134955) [ 4479.944504] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 12:29:18 (1737134958) [ 4482.082381] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 12:29:20 (1737134960) [ 4484.173083] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 12:29:22 (1737134962) [ 4486.690636] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 12:29:24 (1737134964) [ 4488.463707] Lustre: Increasing provided stripe size to a minimum value 64 [ 4491.969431] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 12:29:30 (1737134970) [ 4494.226209] Lustre: *** cfs_fail_loc=168, val=0*** [ 4494.794738] Lustre: *** cfs_fail_loc=168, val=0*** [ 4494.796602] Lustre: Skipped 2 previous similar messages [ 4495.987929] Lustre: *** cfs_fail_loc=168, val=0*** [ 4495.990492] Lustre: Skipped 5 previous similar messages [ 4500.139664] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 12:29:38 (1737134978) [ 4503.430081] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 12:29:41 (1737134981) [ 4505.500654] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 12:29:43 (1737134983) [ 4507.696331] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 12:29:45 (1737134985) [ 4509.842847] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 12:29:47 (1737134987) [ 4511.937059] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 12:29:50 (1737134990) [ 4545.906758] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 12:30:24 (1737135024) [ 4548.467526] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 12:30:26 (1737135026) [ 4551.085998] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 12:30:29 (1737135029) [ 4554.172927] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 12:30:32 (1737135032) [ 4556.525103] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 12:30:34 (1737135034) [ 4559.620960] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 12:30:37 (1737135037) [ 4562.383563] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 12:30:40 (1737135040) [ 4566.432356] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 12:30:44 (1737135044) [ 4570.258125] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 12:30:48 (1737135048) [ 4573.618734] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 12:30:51 (1737135051) [ 4576.466176] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 12:30:54 (1737135054) [ 4576.859919] LustreError: 4987:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 16b sleeping for 2000ms [ 4578.868359] LustreError: 4987:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 16b awake [ 4580.923824] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 12:30:59 (1737135059) [ 4584.864987] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 12:31:02 (1737135062) [ 4585.689035] LustreError: 10638:0:(obd_class.h:1061:obd_statfs()) Device 13 not setup [ 4586.572975] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4588.491688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.498853] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4589.637107] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4590.041427] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4590.294150] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4590.306340] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37473 [ 4591.001727] LustreError: 13034:0:(obd_class.h:1061:obd_statfs()) Device 13 not setup [ 4595.193767] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4595.345391] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37505 [ 4596.567104] LustreError: 15412:0:(obd_class.h:1061:obd_statfs()) Device 13 not setup [ 4596.570129] LustreError: 15412:0:(obd_class.h:1061:obd_statfs()) Skipped 5 previous similar messages [ 4600.898629] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4601.025366] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37537 [ 4602.282672] LustreError: 17819:0:(obd_class.h:1061:obd_statfs()) Device 13 not setup [ 4602.285880] LustreError: 17819:0:(obd_class.h:1061:obd_statfs()) Skipped 7 previous similar messages [ 4606.205660] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4606.631129] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37569 [ 4607.434747] LustreError: 20273:0:(obd_class.h:1061:obd_statfs()) Device 13 not setup [ 4607.438163] LustreError: 20273:0:(obd_class.h:1061:obd_statfs()) Skipped 6 previous similar messages [ 4610.991531] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4611.124837] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37601 [ 4616.823655] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4617.375313] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37633 [ 4618.230100] LustreError: 25009:0:(obd_class.h:1061:obd_statfs()) Device 13 not setup [ 4618.235726] LustreError: 25009:0:(obd_class.h:1061:obd_statfs()) Skipped 17 previous similar messages [ 4622.014021] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4622.387700] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37665 [ 4626.820686] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4626.975480] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37697 [ 4631.186328] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4631.672102] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37729 [ 4635.491126] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4635.912215] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37761 [ 4636.617764] LustreError: 2366:0:(obd_class.h:1061:obd_statfs()) Device 13 not setup [ 4636.620330] LustreError: 2366:0:(obd_class.h:1061:obd_statfs()) Skipped 24 previous similar messages [ 4639.860895] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4640.604409] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37793 [ 4644.276192] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4645.089982] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37825 [ 4648.745731] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4649.474198] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37857 [ 4653.112557] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4653.660732] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37889 [ 4657.374990] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4658.115715] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37921 [ 4661.540575] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4662.601724] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37953 [ 4662.652954] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_create to node 0@lo failed: rc = -19 [ 4662.656219] LustreError: 28259:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -19 [ 4665.746108] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4666.458889] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37953 [ 4669.097956] LustreError: 20514:0:(obd_class.h:1061:obd_statfs()) Device 13 not setup [ 4669.100598] LustreError: 20514:0:(obd_class.h:1061:obd_statfs()) Skipped 63 previous similar messages [ 4670.125007] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4670.737064] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:37985 [ 4674.137511] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4674.777116] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:38017 [ 4678.512939] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4678.725169] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:38049 [ 4681.354150] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 12:32:39 (1737135159) [ 4683.472859] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 12:32:41 (1737135161) [ 4683.914375] Lustre: DEBUG MARKER: SKIP: sanity test_278 needs >= 2 MDTs [ 4684.419857] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 12:32:42 (1737135162) [ 4684.996166] LustreError: 28208:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 15e sleeping [ 4686.629827] LustreError: 27389:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4686.632399] LustreError: 28208:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 15e awake: rc=3366 [ 4687.638172] LustreError: 27389:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4690.206315] LustreError: 166-1: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4690.211586] Lustre: Evicted from MGS (at 192.168.201.138@tcp) after server handle changed from 0x89262a9e1fdf89db to 0x89262a9e1fe298e6 [ 4690.262784] LustreError: 28062:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4691.329155] LustreError: 28063:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4692.333691] LustreError: 28063:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4694.341877] LustreError: 28063:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4694.344809] LustreError: 28063:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message [ 4698.358905] LustreError: 28214:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4698.362048] LustreError: 28214:0:(libcfs_fail.h:180:cfs_race()) Skipped 3 previous similar messages [ 4700.432704] Lustre: lustre-OST0001: deleting orphan objects from 0x0:37367 to 0x0:37409 [ 4700.436973] Lustre: lustre-OST0000: deleting orphan objects from 0x0:37441 to 0x0:38081 [ 4701.474203] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4707.170058] LustreError: 28214:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4707.172798] LustreError: 28214:0:(libcfs_fail.h:180:cfs_race()) Skipped 6 previous similar messages [ 4710.102090] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 12:33:08 (1737135188) [ 4710.550352] Lustre: DEBUG MARKER: SKIP: sanity test_300a needs >= 2 MDTs [ 4711.080042] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 12:33:09 (1737135189) [ 4711.568117] Lustre: DEBUG MARKER: SKIP: sanity test_300b needs >= 2 MDTs [ 4712.112970] Lustre: DEBUG MARKER: == sanity test 300c: chown [ 4712.544659] Lustre: DEBUG MARKER: SKIP: sanity test_300c needs >= 2 MDTs [ 4713.079831] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 12:33:11 (1737135191) [ 4713.539064] Lustre: DEBUG MARKER: SKIP: sanity test_300d needs >= 2 MDTs [ 4714.056655] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ========================================================== 12:33:12 (1737135192) [ 4714.504891] Lustre: DEBUG MARKER: SKIP: sanity test_300e needs >= 2 MDTs [ 4715.039095] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ========================================================== 12:33:13 (1737135193) [ 4715.557019] Lustre: DEBUG MARKER: SKIP: sanity test_300f needs >= 2 MDTs [ 4716.142452] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ========================================================== 12:33:14 (1737135194) [ 4716.633933] Lustre: DEBUG MARKER: SKIP: sanity test_300g needs >= 2 MDTs [ 4717.210226] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory ========================================================== 12:33:15 (1737135195) [ 4717.730310] Lustre: DEBUG MARKER: SKIP: sanity test_300h needs >= 2 MDTs [ 4718.331004] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory ========================================================== 12:33:16 (1737135196) [ 4718.798005] Lustre: DEBUG MARKER: SKIP: sanity test_300i needs >= 2 MDTs [ 4719.313516] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ============ 12:33:17 (1737135197) [ 4719.781491] Lustre: DEBUG MARKER: SKIP: sanity test_300j needs >= 2 MDTs [ 4720.327223] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ======== 12:33:18 (1737135198) [ 4720.763989] Lustre: DEBUG MARKER: SKIP: sanity test_300k needs >= 2 MDTs [ 4721.285846] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ========================================================== 12:33:19 (1737135199) [ 4721.752052] Lustre: DEBUG MARKER: SKIP: sanity test_300l needs >= 2 MDTs [ 4722.284583] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS ========================================================== 12:33:20 (1737135200) [ 4724.409173] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA ========================================================== 12:33:22 (1737135202) [ 4724.907495] Lustre: DEBUG MARKER: SKIP: sanity test_300n needs >= 2 MDTs [ 4725.569224] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [ 4726.158132] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================================== 12:33:24 (1737135204) [ 4726.668389] Lustre: DEBUG MARKER: SKIP: sanity test_300p needs >= 2 MDTs [ 4727.253091] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ========================================================== 12:33:25 (1737135205) [ 4727.717780] Lustre: DEBUG MARKER: SKIP: sanity test_300q needs >= 2 MDTs [ 4728.395380] Lustre: DEBUG MARKER: == sanity test 300r: test -1 striped directory =========== 12:33:26 (1737135206) [ 4728.971742] Lustre: DEBUG MARKER: SKIP: sanity test_300r needs >= 2 MDTs [ 4729.650892] Lustre: DEBUG MARKER: == sanity test 300s: test lfs mkdir -c without -i ======== 12:33:27 (1737135207) [ 4730.214334] Lustre: DEBUG MARKER: SKIP: sanity test_300s needs >= 2 MDTs [ 4730.861329] Lustre: DEBUG MARKER: == sanity test 300t: test max_mdt_stripecount ============ 12:33:28 (1737135208) [ 4731.413200] Lustre: DEBUG MARKER: SKIP: sanity test_300t needs at least 2 MDTs [ 4732.056018] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ============= 12:33:30 (1737135210) [ 4732.629087] Lustre: DEBUG MARKER: SKIP: sanity test_310a needs >= 4 MDTs [ 4733.307864] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open ========================================================== 12:33:31 (1737135211) [ 4733.930884] Lustre: DEBUG MARKER: SKIP: sanity test_310b needs >= 4 MDTs [ 4734.677156] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ========================================================== 12:33:32 (1737135212) [ 4735.213989] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [ 4735.837986] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ========================================================== 12:33:33 (1737135213) [ 4752.436422] Lustre: DEBUG MARKER: == sanity test 312: make sure ZFS adjusts its block size by write pattern ========================================================== 12:33:50 (1737135230) [ 4814.581531] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ========================================================== 12:34:52 (1737135292) [ 4814.976218] Lustre: *** cfs_fail_loc=720, val=0*** [ 4814.977687] LustreError: 1146:0:(ofd_objects.c:987:ofd_object_punch()) lustre-OST0000: failed to stop transaction: rc = -5 [ 4817.122070] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ========================================================== 12:34:55 (1737135295) [ 4818.081022] Lustre: *** cfs_fail_loc=720, val=0*** [ 4818.081032] LustreError: 16402:0:(ofd_objects.c:1054:ofd_destroy()) lustre-OST0001 failed to stop transaction: -5 [ 4818.081051] LustreError: 16402:0:(ofd_dev.c:1837:ofd_destroy_hdl()) lustre-OST0001: error destroying object [0x100010000:0x9225:0x0]: -5 [ 4818.090943] Lustre: Skipped 1 previous similar message [ 4840.519242] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ============= 12:35:18 (1737135318) [ 4840.558373] LustreError: 1147:0:(tgt_grant.c:767:tgt_grant_check()) lustre-OST0000: cli a784e15b-d0a7-40ec-a2e5-f219489e18dd claims 1703936 GRANT, real grant 0 [ 4845.630298] Lustre: DEBUG MARKER: == sanity test 316: lfs migrate of file with large_xattr enabled ========================================================== 12:35:23 (1737135323) [ 4846.525824] Lustre: DEBUG MARKER: SKIP: sanity test_316 needs >= 2 MDTs [ 4847.329890] hrtimer: interrupt took 4614641 ns [ 4847.801388] Lustre: DEBUG MARKER: == sanity test 317: Verify blocks get correctly update after truncate ========================================================== 12:35:25 (1737135325) [ 4848.938805] Lustre: DEBUG MARKER: SKIP: sanity test_317 LU-10370: no implementation for ZFS [ 4850.525583] Lustre: DEBUG MARKER: == sanity test 318: Verify async readahead tunables ====== 12:35:27 (1737135327) [ 4854.432134] Lustre: DEBUG MARKER: == sanity test 319: lost lease lock on migrate error ===== 12:35:32 (1737135332) [ 4855.527313] Lustre: DEBUG MARKER: SKIP: sanity test_319 needs >= 2 MDTs [ 4856.802169] Lustre: DEBUG MARKER: == sanity test 350: force NID mismatch path to be exercised ========================================================== 12:35:34 (1737135334) [ 4857.734877] LNet: *** cfs_fail_loc=e001, val=0*** [ 4857.738571] LNet: 31091:0:(lib-ptl.c:184:lnet_try_match_md()) message from 12345-192.168.201.38@tcp matched 1821515825983040 with NID mismatch 192.168.201.38@tcp accepted (same peer ffff88009c40aa00) [ 4858.256244] LNet: *** cfs_fail_loc=e001, val=0*** [ 4858.257761] LNet: Skipped 23 previous similar messages [ 4858.259323] LNet: 31091:0:(lib-ptl.c:184:lnet_try_match_md()) message from 12345-192.168.201.38@tcp matched 1821515825984576 with NID mismatch 192.168.201.38@tcp accepted (same peer ffff88009c40aa00) [ 4858.268213] LNet: 31091:0:(lib-ptl.c:184:lnet_try_match_md()) Skipped 23 previous similar messages [ 4859.288159] LNet: *** cfs_fail_loc=e001, val=0*** [ 4859.292732] LNet: Skipped 28 previous similar messages [ 4859.296547] LNet: 31091:0:(lib-ptl.c:184:lnet_try_match_md()) message from 12345-192.168.201.38@tcp matched 1821515825986432 with NID mismatch 192.168.201.38@tcp accepted (same peer ffff88009c40aa00) [ 4859.305019] LNet: 31091:0:(lib-ptl.c:184:lnet_try_match_md()) Skipped 28 previous similar messages [ 4974.345384] Lustre: DEBUG MARKER: == sanity test 398a: direct IO should cancel lock otherwise lockless ========================================================== 12:37:32 (1737135452) [ 4977.176580] Lustre: DEBUG MARKER: == sanity test 398b: DIO and buffer IO race ============== 12:37:35 (1737135455) [ 5002.958506] Lustre: DEBUG MARKER: == sanity test 398c: run fio to test AIO ================= 12:38:01 (1737135481) [ 5125.878752] Lustre: DEBUG MARKER: == sanity test 398d: run aiocp to verify block size > stripe size ========================================================== 12:40:03 (1737135603) [ 5131.132564] Lustre: DEBUG MARKER: == sanity test 398e: O_Direct open cleared by fcntl doesn't cause hang ========================================================== 12:40:09 (1737135609) [ 5134.193953] Lustre: DEBUG MARKER: == sanity test 398f: verify aio handles ll_direct_rw_pages errors correctly ========================================================== 12:40:12 (1737135612) [ 5137.362959] Lustre: DEBUG MARKER: == sanity test 398g: verify parallel dio async RPC submission ========================================================== 12:40:15 (1737135615) [ 5138.115670] LustreError: 1146:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5138.119774] LustreError: 1146:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 5140.117537] LustreError: 29633:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 5140.122256] LustreError: 29633:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 5140.431738] LustreError: 1146:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5140.435198] LustreError: 1146:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 5 previous similar messages [ 5142.505486] LustreError: 1146:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5144.509373] LustreError: 1146:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 5144.511749] LustreError: 1146:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 7 previous similar messages [ 5146.640448] LustreError: 1146:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5146.648114] LustreError: 1146:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 5152.782414] LustreError: 1146:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 5152.785778] LustreError: 1146:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 5154.974443] LustreError: 1146:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5154.979447] LustreError: 1146:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 5160.294194] Lustre: DEBUG MARKER: == sanity test 398h: verify correctness of read [ 5166.604170] Lustre: DEBUG MARKER: == sanity test 398i: verify parallel dio handles ll_direct_rw_pages errors correctly ========================================================== 12:40:44 (1737135644) [ 5170.584538] Lustre: DEBUG MARKER: == sanity test 398j: test parallel dio where stripe size > rpc_size ========================================================== 12:40:48 (1737135648) [ 5176.176227] Lustre: DEBUG MARKER: == sanity test 398k: test enospc on first stripe ========= 12:40:54 (1737135654) [ 5196.182976] Lustre: DEBUG MARKER: SKIP: sanity test_398k 7379968 > 600000 skipping out-of-space test on OST0 [ 5196.699552] Lustre: DEBUG MARKER: == sanity test 398l: test enospc on intermediate stripe/RPC ========================================================== 12:41:14 (1737135674) [ 5207.095488] Lustre: DEBUG MARKER: SKIP: sanity test_398l 7359488 > 600000 skipping out-of-space test on OST0 [ 5207.714488] Lustre: DEBUG MARKER: == sanity test 398m: test RPC failures with parallel dio ========================================================== 12:41:25 (1737135685) [ 5208.156977] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5208.159291] Lustre: Skipped 7 previous similar messages [ 5209.186968] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5209.189388] Lustre: Skipped 3 previous similar messages [ 5211.211010] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5211.212718] Lustre: Skipped 3 previous similar messages [ 5214.000479] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5214.000480] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5214.004327] Lustre: Skipped 1 previous similar message [ 5218.017363] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5218.019828] Lustre: Skipped 3 previous similar messages [ 5229.039984] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5229.042823] Lustre: Skipped 7 previous similar messages [ 5253.073747] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5253.075788] Lustre: Skipped 11 previous similar messages [ 5286.100795] Lustre: *** cfs_fail_loc=20f, val=0*** [ 5286.102565] Lustre: Skipped 35 previous similar messages [ 5356.887749] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5356.887751] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5356.887755] Lustre: Skipped 51 previous similar messages [ 5435.740581] Lustre: DEBUG MARKER: == sanity test 398n: test append with parallel DIO ======= 12:45:13 (1737135913) [ 5439.241317] Lustre: DEBUG MARKER: == sanity test 399a: fake write should not be slower than normal write ========================================================== 12:45:17 (1737135917) [ 5467.951085] Lustre: *** cfs_fail_loc=238, val=0*** [ 5500.451011] Lustre: DEBUG MARKER: sanity test_399a: @@@@@@ IGNORE (env=kvm): fake write is slower