[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] mapped at [ffffffffff200b50] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2f2000-0xbffcffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5970 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 276616662 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffd9fff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043832 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782298 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027499 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820316k/5224448k available (8172k kernel code, 1049120k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.389045] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.390736] pid_max: default: 32768 minimum: 301 [ 0.391721] Security Framework initialized [ 0.392509] SELinux: Initializing. [ 0.393254] SELinux: Starting in permissive mode [ 0.394575] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.397779] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.401026] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.403596] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.405281] Initializing cgroup subsys memory [ 0.406374] Initializing cgroup subsys devices [ 0.407804] Initializing cgroup subsys freezer [ 0.408704] Initializing cgroup subsys net_cls [ 0.409767] Initializing cgroup subsys blkio [ 0.410676] Initializing cgroup subsys perf_event [ 0.411647] Initializing cgroup subsys hugetlb [ 0.412581] Initializing cgroup subsys pids [ 0.413511] Initializing cgroup subsys net_prio [ 0.414671] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.417009] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.418283] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.419311] tlb_flushall_shift: 6 [ 0.420098] FEATURE SPEC_CTRL Present [ 0.420980] FEATURE IBPB_SUPPORT Present [ 0.421796] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.423190] Spectre V2 : Vulnerable [ 0.424008] Speculative Store Bypass: Vulnerable [ 0.426382] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.432863] ACPI: Core revision 20130517 [ 0.435381] ACPI: All ACPI Tables successfully acquired [ 0.436868] ftrace: allocating 30294 entries in 119 pages [ 0.481915] Enabling x2apic [ 0.482560] Enabled x2apic [ 0.483398] Switched APIC routing to physical x2apic. [ 0.485767] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.487114] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.491212] TSC deadline timer enabled [ 0.491422] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.495096] ... version: 2 [ 0.496748] ... bit width: 48 [ 0.498553] ... generic registers: 4 [ 0.500006] ... value mask: 0000ffffffffffff [ 0.501003] ... max period: 00007fffffffffff [ 0.502027] ... fixed-purpose events: 3 [ 0.502788] ... event mask: 000000070000000f [ 0.503888] KVM setup paravirtual spinlock [ 0.508083] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.511598] KVM setup async PF for cpu 1 [ 0.513033] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.515518] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.517988] KVM setup async PF for cpu 2 [ 0.506649] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.518558] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.520320] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.521361] Brought up 4 CPUs [ 0.521384] KVM setup async PF for cpu 3 [ 0.521392] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.523815] smpboot: Max logical packages: 1 [ 0.524646] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.528570] devtmpfs: initialized [ 0.529453] x86/mm: Memory block size: 128MB [ 0.533169] EVM: security.selinux [ 0.533796] EVM: security.ima [ 0.534606] EVM: security.capability [ 0.537241] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.539146] NET: Registered protocol family 16 [ 0.541415] cpuidle: using governor haltpoll [ 0.543461] ACPI: bus type PCI registered [ 0.545200] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.548017] PCI: Using configuration type 1 for base access [ 0.550102] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.558325] ACPI: Added _OSI(Module Device) [ 0.559927] ACPI: Added _OSI(Processor Device) [ 0.560845] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.561773] ACPI: Added _OSI(Processor Aggregator Device) [ 0.562891] ACPI: Added _OSI(Linux-Dell-Video) [ 0.564567] ACPI: EC: Look up EC in DSDT [ 0.566557] ACPI: Interpreter enabled [ 0.567334] ACPI: (supports S0 S3 S4 S5) [ 0.568273] ACPI: Using IOAPIC for interrupt routing [ 0.569248] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.571402] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.576844] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.578141] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.579654] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.581129] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.584049] acpiphp: Slot [2] registered [ 0.584949] acpiphp: Slot [3] registered [ 0.585934] acpiphp: Slot [4] registered [ 0.586783] acpiphp: Slot [5] registered [ 0.587643] acpiphp: Slot [6] registered [ 0.588601] acpiphp: Slot [7] registered [ 0.589519] acpiphp: Slot [8] registered [ 0.590337] acpiphp: Slot [9] registered [ 0.591277] acpiphp: Slot [10] registered [ 0.592218] acpiphp: Slot [11] registered [ 0.593081] acpiphp: Slot [12] registered [ 0.593915] acpiphp: Slot [13] registered [ 0.594827] acpiphp: Slot [14] registered [ 0.595770] acpiphp: Slot [15] registered [ 0.596666] acpiphp: Slot [16] registered [ 0.597533] acpiphp: Slot [17] registered [ 0.598381] acpiphp: Slot [18] registered [ 0.599332] acpiphp: Slot [19] registered [ 0.600255] acpiphp: Slot [20] registered [ 0.601133] acpiphp: Slot [21] registered [ 0.602058] acpiphp: Slot [22] registered [ 0.602970] acpiphp: Slot [23] registered [ 0.603849] acpiphp: Slot [24] registered [ 0.604719] acpiphp: Slot [25] registered [ 0.605693] acpiphp: Slot [26] registered [ 0.606613] acpiphp: Slot [27] registered [ 0.607455] acpiphp: Slot [28] registered [ 0.608364] acpiphp: Slot [29] registered [ 0.609276] acpiphp: Slot [30] registered [ 0.610138] acpiphp: Slot [31] registered [ 0.611041] PCI host bridge to bus 0000:00 [ 0.611888] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.613242] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.614680] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.616148] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.617824] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.619332] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.620608] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.621161] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.621820] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.625472] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.627298] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.628981] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.630414] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.631842] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.633865] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.634604] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.636877] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.639913] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.641589] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.647288] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.648832] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.649708] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.652327] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.654597] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.664231] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.667268] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.669298] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.671086] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.677449] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.681791] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.684000] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.685686] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.687032] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.688426] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.690401] vgaarb: loaded [ 0.691512] SCSI subsystem initialized [ 0.692327] ACPI: bus type USB registered [ 0.693146] usbcore: registered new interface driver usbfs [ 0.694358] usbcore: registered new interface driver hub [ 0.695510] usbcore: registered new device driver usb [ 0.697742] PCI: Using ACPI for IRQ routing [ 0.698604] PCI: pci_cache_line_size set to 64 bytes [ 0.698734] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.698737] e820: reserve RAM buffer [mem 0xbffda000-0xbfffffff] [ 0.698739] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 0.699010] NetLabel: Initializing [ 0.699792] NetLabel: domain hash size = 128 [ 0.700801] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.701784] NetLabel: unlabeled traffic allowed by default [ 0.703158] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.704178] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.707519] amd_nb: Cannot enumerate AMD northbridges [ 0.708557] Switched to clocksource kvm-clock [ 0.724728] pnp: PnP ACPI init [ 0.726023] ACPI: bus type PNP registered [ 0.727569] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.727630] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.727663] pnp 00:02: [dma 2] [ 0.727693] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.727750] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.727806] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.727861] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.728149] pnp: PnP ACPI: found 6 devices [ 0.729002] ACPI: bus type PNP unregistered [ 0.738956] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.738962] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.738964] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.738966] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.738968] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 0.739031] NET: Registered protocol family 2 [ 0.740337] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 0.742052] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 0.744359] TCP: Hash tables configured (established 32768 bind 32768) [ 0.745739] TCP: reno registered [ 0.746416] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 0.747689] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 0.749192] NET: Registered protocol family 1 [ 0.750773] RPC: Registered named UNIX socket transport module. [ 0.752823] RPC: Registered udp transport module. [ 0.754371] RPC: Registered tcp transport module. [ 0.755881] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.758100] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.759508] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.760955] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.762373] PCI: CLS 0 bytes, default 64 [ 0.762532] Unpacking initramfs... [ 2.140082] debug: unmapping init [mem 0xffff8800bc2f2000-0xffff8800bffcffff] [ 2.144294] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.146385] software IO TLB [mem 0xb82f2000-0xbc2f2000] (64MB) mapped at [ffff8800b82f2000-ffff8800bc2f1fff] [ 2.149621] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.152268] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.154160] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.156590] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.161225] cryptomgr_test (51) used greatest stack depth: 14128 bytes left [ 2.161684] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.161744] Initialise system trusted keyring [ 2.194543] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.197249] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.203662] zpool: loaded [ 2.204591] zbud: loaded [ 2.205757] VFS: Disk quotas dquot_6.6.0 [ 2.206951] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.209377] NFS: Registering the id_resolver key type [ 2.210751] Key type id_resolver registered [ 2.212186] Key type id_legacy registered [ 2.213773] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.215851] Key type big_key registered [ 2.216675] SELinux: Registering netfilter hooks [ 2.217864] cryptomgr_test (57) used greatest stack depth: 14048 bytes left [ 2.220496] cryptomgr_test (58) used greatest stack depth: 13968 bytes left [ 2.222640] NET: Registered protocol family 38 [ 2.222651] Key type asymmetric registered [ 2.222656] Asymmetric key parser 'x509' registered [ 2.222768] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.222847] io scheduler noop registered [ 2.222852] io scheduler deadline registered (default) [ 2.222920] io scheduler cfq registered [ 2.222925] io scheduler mq-deadline registered [ 2.222929] io scheduler kyber registered [ 2.225293] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.225305] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.240728] intel_idle: does not run on family 6 model 62 [ 2.240857] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.242330] ACPI: Power Button [PWRF] [ 2.243540] GHES: HEST is not enabled! [ 2.304695] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.371458] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.435969] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.466664] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 2.495288] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 2.498256] Non-volatile memory driver v1.3 [ 2.499418] Linux agpgart interface v0.103 [ 2.500687] crash memory driver: version 1.1 [ 2.502960] nbd: registered device at major 43 [ 2.508957] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 2.509000] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 2.509045] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 2.509092] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 2.509143] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 2.516444] virtio_blk virtio1: [vda] 53424 512-byte logical blocks (27.3 MB/26.0 MiB) [ 2.525946] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 2.526001] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 2.526049] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 2.526097] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 2.526154] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 2.533662] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.539498] rdac: device handler registered [ 2.541101] hp_sw: device handler registered [ 2.542152] emc: device handler registered [ 2.543422] libphy: Fixed MDIO Bus: probed [ 2.546808] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 2.548540] ehci-pci: EHCI PCI platform driver [ 2.549510] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 2.552036] ohci-pci: OHCI PCI platform driver [ 2.553866] uhci_hcd: USB Universal Host Controller Interface driver [ 2.556690] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.561539] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.563575] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.566766] mousedev: PS/2 mouse device common for all mice [ 2.569716] rtc_cmos 00:05: RTC can wake from S4 [ 2.570380] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.577708] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 2.581399] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.584328] hidraw: raw HID events driver (C) Jiri Kosina [ 2.585898] usbcore: registered new interface driver usbhid [ 2.588035] usbhid: USB HID core driver [ 2.589743] drop_monitor: Initializing network drop monitor service [ 2.592515] Netfilter messages via NETLINK v0.30. [ 2.594723] TCP: cubic registered [ 2.596110] Initializing XFRM netlink socket [ 2.597987] NET: Registered protocol family 10 [ 2.599640] NET: Registered protocol family 17 [ 2.600915] Key type dns_resolver registered [ 2.602507] mce: Using 10 MCE banks [ 2.603933] Loading compiled-in X.509 certificates [ 2.606236] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 2.610230] registered taskstats version 1 [ 2.614100] modprobe (72) used greatest stack depth: 13456 bytes left [ 2.618578] Key type trusted registered [ 2.625535] Key type encrypted registered [ 2.627314] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 2.631318] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 2.635949] rtc_cmos 00:05: setting system clock to 2025-01-16 01:21:19 UTC (1736990479) [ 2.639857] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 2.642820] Write protecting the kernel read-only data: 12288k [ 2.645608] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 2.648389] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 2.659249] random: systemd: uninitialized urandom read (16 bytes read) [ 2.664012] random: systemd: uninitialized urandom read (16 bytes read) [ 2.667073] random: systemd: uninitialized urandom read (16 bytes read) [ 2.672426] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 2.680022] systemd[1]: Detected virtualization kvm. [ 2.681923] systemd[1]: Detected architecture x86-64. [ 2.683807] systemd[1]: Running in initial RAM disk. [ 2.688767] systemd[1]: No hostname configured. [ 2.690210] systemd[1]: Set hostname to . [ 2.691503] random: systemd: uninitialized urandom read (16 bytes read) [ 2.693805] systemd[1]: Initializing machine ID from random generator. [ 2.746205] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 2.751530] random: systemd: uninitialized urandom read (16 bytes read) [ 2.754136] random: systemd: uninitialized urandom read (16 bytes read) [ 2.756882] random: systemd: uninitialized urandom read (16 bytes read) [ 2.759364] random: systemd: uninitialized urandom read (16 bytes read) [ 2.762591] random: systemd: uninitialized urandom read (16 bytes read) [ 2.764660] random: systemd: uninitialized urandom read (16 bytes read) [ 2.775017] systemd[1]: Reached target Timers. [ 2.779379] systemd[1]: Reached target Swap. [ 2.783971] systemd[1]: Reached target Local File Systems. [ 2.789202] systemd[1]: Created slice Root Slice. [ 2.792815] systemd[1]: Listening on udev Kernel Socket. [ 2.796844] systemd[1]: Listening on udev Control Socket. [ 2.800815] systemd[1]: Created slice System Slice. [ 2.804723] systemd[1]: Reached target Slices. [ 2.808746] systemd[1]: Listening on Journal Socket. [ 2.813510] systemd[1]: Starting Setup Virtual Console... [ 2.818042] systemd[1]: Reached target Sockets. [ 2.823704] systemd[1]: Starting Journal Service... [ 2.830212] systemd[1]: Starting Load Kernel Modules... [ 2.836443] systemd[1]: Starting dracut cmdline hook... [ 2.843063] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 2.851677] systemd[1]: Started Setup Virtual Console. [ 2.859487] systemd[1]: Started Load Kernel Modules. [ 2.864765] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 2.872850] systemd[1]: Started Journal Service. [ 2.982975] random: fast init done [ 3.161655] tsc: Refined TSC clocksource calibration: 2399.947 MHz [ 3.400498] libata version 3.00 loaded. [ 3.405032] ata_piix 0000:00:01.1: version 2.13 [ 3.422507] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 3.428808] scsi host0: ata_piix [ 3.432759] scsi host1: ata_piix [ 3.435462] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 3.437508] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 3.631701] ip (320) used greatest stack depth: 13080 bytes left [ 3.690093] ip (343) used greatest stack depth: 12336 bytes left [ 6.719266] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 7.149090] systemd-journald[102]: Received SIGTERM from PID 1 (systemd). [ 7.375271] SELinux: Disabled at runtime. [ 7.376334] SELinux: Unregistering netfilter hooks [ 7.459005] ip_tables: (C) 2000-2006 Netfilter Core Team [ 7.463026] systemd[1]: Inserted module 'ip_tables' [ 7.859035] systemd-journald[562]: Received request to flush runtime journal from PID 1 [ 8.009783] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 8.024037] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 8.064278] cryptd: max_cpu_qlen set to 1000 [ 8.108068] AVX version of gcm_enc/dec engaged. [ 8.109855] AES CTR mode by8 optimization enabled [ 8.132891] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 8.142618] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 8.156985] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 8.161899] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 8.266285] EDAC MC: Ver: 3.0.0 [ 8.273977] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 8.273988] EDAC sbridge: Ver: 1.1.2 [ 10.359088] mount.nfs (764) used greatest stack depth: 10704 bytes left [ 27.489010] libcfs: loading out-of-tree module taints kernel. [ 27.491102] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 27.509732] alg: No test for adler32 (adler32-zlib) [ 28.352575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 28.493365] Lustre: Lustre: Build Version: 2.15.6_6_ga1ea412 [ 28.664544] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 28.666168] LNet: Accept secure, port 988 [ 30.210658] Key type lgssc registered [ 30.510818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.888370] Lustre: Mounted lustre-client [ 79.900602] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 90.006896] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing check_logdir /tmp/testlogs/ [ 91.088335] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing yml_node [ 92.126499] Lustre: DEBUG MARKER: Client: 2.15.6.6 [ 92.806630] Lustre: DEBUG MARKER: MDS: 2.15.6.6 [ 93.495116] Lustre: DEBUG MARKER: OSS: 2.15.6.6 [ 93.930641] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Wed Jan 15 20:22:50 EST 2025 [ 95.647539] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42b 42c 407 118c 118d [ 96.035754] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 115 135 136 230d 300o [ 97.471881] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing check_config_client /mnt/lustre [ 102.656689] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 102.917785] Lustre: lustre-OST0000-osc-ffff88012a485000: disconnect after 23s idle [ 106.488395] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 20:23:02 (1736990582) [ 107.506244] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 107.974894] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 20:23:04 (1736990584) [ 108.026920] dmesg (10254) used greatest stack depth: 10048 bytes left [ 109.921087] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 20:23:06 (1736990586) [ 111.889221] random: crng init done [ 127.957830] Lustre: lustre-OST0001-osc-ffff88012a485000: disconnect after 23s idle [ 127.960508] Lustre: Skipped 1 previous similar message [ 128.472788] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 20:23:24 (1736990604) [ 128.499032] Lustre: DEBUG MARKER: test message ID 25797 7440 [ 130.036902] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 20:23:26 (1736990606) [ 132.078791] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 20:23:28 (1736990608) [ 132.206946] LustreError: dumping log to /tmp/f60f.sanity.1736990609.13084 [ 133.876392] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 20:23:30 (1736990610) [ 133.943366] lfs (13907) used greatest stack depth: 10016 bytes left [ 160.307989] Lustre: dir [0x240000402:0x137:0x0] stripe 0 readdir failed: -2, directory is partially accessed! [ 162.032133] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 20:23:58 (1736990638) [ 162.393631] Lustre: dir [0x240000402:0x275:0x0] stripe 2 readdir failed: -2, directory is partially accessed! [ 162.397515] Lustre: Skipped 1 previous similar message [ 164.783575] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 165.165407] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 20:24:01 (1736990641) [ 166.734029] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 20:24:03 (1736990643) [ 168.277135] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 20:24:04 (1736990644) [ 230.637384] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 20:25:07 (1736990707) [ 230.960049] Lustre: *** cfs_fail_loc=406, val=0*** [ 230.961578] LustreError: 20330:0:(osc_request.c:2750:osc_build_rpc()) prep_req failed: -12 [ 230.964332] LustreError: 20330:0:(osc_cache.c:2186:osc_check_rpcs()) Write request failed with -12 [ 235.241212] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 20:25:11 (1736990711) [ 237.221050] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 237.642052] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 20:25:14 (1736990714) [ 239.548096] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 20:25:16 (1736990716) [ 254.498263] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 20:25:31 (1736990731) [ 254.731690] Lustre: Unmounted lustre-client [ 254.811431] Lustre: Mounted lustre-client [ 255.563310] Lustre: Unmounted lustre-client [ 257.653473] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 20:25:34 (1736990734) [ 257.840703] Lustre: Unmounted lustre-client [ 257.918505] Lustre: Mounted lustre-client [ 257.919400] Lustre: Skipped 1 previous similar message [ 259.889713] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 20:25:36 (1736990736) [ 297.155035] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 20:26:13 (1736990773) [ 307.310264] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 20:26:23 (1736990783) [ 313.383832] Lustre: lustre-OST0000-osc-ffff8800a9d1d000: Connection to lustre-OST0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 316.230665] Lustre: 1878:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736990786/real 1736990786] req@ffff8800a8538980 x1821366558463296/t0(0) o17->lustre-OST0000-osc-ffff8800a9d1d000@192.168.203.137@tcp:28/4 lens 456/432 e 0 to 1 dl 1736990793 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'lctl.0' [ 324.064875] Lustre: lustre-OST0000-osc-ffff8800a9d1d000: Connection restored to (at 192.168.203.137@tcp) [ 325.786800] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 326.159770] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 329.336646] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 20:26:45 (1736990805) [ 330.874985] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 20:26:47 (1736990807) [ 332.499309] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 20:26:49 (1736990809) [ 334.106804] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 20:26:50 (1736990810) [ 335.763242] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 20:26:52 (1736990812) [ 337.395102] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 20:26:53 (1736990813) [ 338.907736] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 20:26:55 (1736990815) [ 340.440954] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 20:26:56 (1736990816) [ 342.007398] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 20:26:58 (1736990818) [ 343.658681] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 20:27:00 (1736990820) [ 346.134082] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 20:27:02 (1736990822) [ 379.612538] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 20:27:36 (1736990856) [ 381.189211] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 20:27:37 (1736990857) [ 382.703646] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 20:27:39 (1736990859) [ 396.075429] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 20:27:52 (1736990872) [ 399.029265] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 20:27:55 (1736990875) [ 402.201946] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 402.596427] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 20:27:59 (1736990879) [ 404.272432] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 20:28:00 (1736990880) [ 406.238680] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 20:28:02 (1736990882) [ 433.951593] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 20:28:30 (1736990910) [ 435.520047] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 20:28:32 (1736990912) [ 437.004825] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 20:28:33 (1736990913) [ 437.040226] Lustre: *** cfs_fail_loc=319, val=0*** [ 438.492567] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 20:28:35 (1736990915) [ 448.097751] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 20:28:44 (1736990924) [ 455.886452] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 20:28:52 (1736990932) [ 457.769177] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 20:28:54 (1736990934) [ 457.934784] Lustre: *** cfs_fail_loc=409, val=0*** [ 457.963399] LustreError: 132-0: lustre-OST0000-osc-ffff8800a9d1d000: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 192.168.203.137@tcp inode [0x200000407:0xc14:0x0] object 0x0:3883 extent [0-4194303], original client csum cbad0603 (type 4), server csum cbad0602 (type 4), client csum now cbad0602 [ 457.970271] LustreError: 1878:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800ab0eda40 x1821366560635200/t8589937130(8589937130) o4->lustre-OST0000-osc-ffff8800a9d1d000@192.168.203.137@tcp:6/4 lens 488/448 e 0 to 0 dl 1736990941 ref 3 fl Interpret:RQU/4/0 rc 0/0 job:'dd.0' [ 459.053005] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 459.158965] Lustre: *** cfs_fail_loc=408, val=0*** [ 459.169520] LustreError: 133-1: lustre-OST0000-osc-ffff8800a9d1d000: BAD READ CHECKSUM: from 192.168.203.137@tcp inode [0x200000407:0xc14:0x0] object 0x0:3883 extent [0-4194303], client 5e133886/5e133886, server d918dddc, cksum_type 1 [ 459.175509] LustreError: 1878:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800a9f8a600 x1821366560636416/t0(0) o3->lustre-OST0000-osc-ffff8800a9d1d000@192.168.203.137@tcp:6/4 lens 488/440 e 0 to 0 dl 1736990943 ref 2 fl Interpret:RMQU/0/0 rc 4194304/4194304 job:'cmp.0' [ 460.815181] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 460.926391] LustreError: 133-1: lustre-OST0000-osc-ffff8800a9d1d000: BAD READ CHECKSUM: from 192.168.203.137@tcp inode [0x200000407:0xc14:0x0] object 0x0:3883 extent [0-4194303], client 3d16ed63/3d16ed63, server f35dedd8, cksum_type 2 [ 460.931748] LustreError: 1875:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880136a82ac0 x1821366560637184/t0(0) o3->lustre-OST0000-osc-ffff8800a9d1d000@192.168.203.137@tcp:6/4 lens 488/440 e 0 to 0 dl 1736990944 ref 2 fl Interpret:RMQU/0/0 rc 4194304/4194304 job:'cmp.0' [ 462.569868] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 462.657667] Lustre: *** cfs_fail_loc=408, val=0*** [ 462.659226] Lustre: Skipped 1 previous similar message [ 462.665727] LustreError: 133-1: lustre-OST0000-osc-ffff8800a9d1d000: BAD READ CHECKSUM: from 192.168.203.137@tcp inode [0x200000407:0xc14:0x0] object 0x0:3883 extent [0-4194303], client cf4d6c96/cf4d6c96, server cbad0602, cksum_type 4 [ 464.286375] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 464.385480] LustreError: 1875:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880136a83440 x1821366560638976/t0(0) o3->lustre-OST0000-osc-ffff8800a9d1d000@192.168.203.137@tcp:6/4 lens 488/440 e 0 to 0 dl 1736990948 ref 2 fl Interpret:RMQU/0/0 rc 4194304/4194304 job:'cmp.0' [ 464.391204] LustreError: 1875:0:(osc_request.c:2397:osc_brw_redo_request()) Skipped 1 previous similar message [ 466.045694] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 466.126030] LustreError: 133-1: lustre-OST0000-osc-ffff8800a9d1d000: BAD READ CHECKSUM: from 192.168.203.137@tcp inode [0x200000407:0xc14:0x0] object 0x0:3883 extent [0-4194303], client eeeaeff8/eeeaeff8, server 46bbef83, cksum_type 20 [ 466.130258] LustreError: Skipped 1 previous similar message [ 467.787456] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 467.891072] Lustre: *** cfs_fail_loc=408, val=0*** [ 467.892337] Lustre: Skipped 2 previous similar messages [ 469.282839] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 469.395015] LustreError: 1876:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880136a83440 x1821366560640640/t0(0) o3->lustre-OST0000-osc-ffff8800a9d1d000@192.168.203.137@tcp:6/4 lens 488/440 e 0 to 0 dl 1736990953 ref 2 fl Interpret:RMQU/0/0 rc 4194304/4194304 job:'cmp.0' [ 469.400330] LustreError: 1876:0:(osc_request.c:2397:osc_brw_redo_request()) Skipped 2 previous similar messages [ 470.965467] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 472.557078] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 20:29:09 (1736990949) [ 473.451035] Lustre: 1878:0:(osc_request.c:1975:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-osc-[0x200000407:0xc15:0x0]:[0-1048575]-9f0c55c9-a46bae53 [ 473.456464] LustreError: dumping log to /tmp/lustre-log.1736990950.1878 [ 473.474754] LustreError: 133-1: lustre-OST0000-osc-ffff8800a9d1d000: BAD READ CHECKSUM: from 192.168.203.137@tcp inode [0x200000407:0xc15:0x0] object 0x0:3884 extent [0-1048575], client 9f0c55c9/9f0c55c9, server a46bae53, cksum_type 4 [ 473.479869] LustreError: Skipped 2 previous similar messages [ 483.644048] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 20:29:20 (1736990960) [ 483.746990] Lustre: *** cfs_fail_loc=409, val=0*** [ 483.748542] Lustre: Skipped 2 previous similar messages [ 483.785149] LustreError: 132-0: lustre-OST0001-osc-ffff8800a9d1d000: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 192.168.203.137@tcp inode [0x200000407:0xc17:0x0] object 0x0:3818 extent [0-4194303], original client csum de2bf73f (type 4), server csum de2bf73e (type 4), client csum now de2bf73e [ 483.793382] LustreError: 1875:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88007e3bb900 x1821366560644992/t4294972506(4294972506) o4->lustre-OST0001-osc-ffff8800a9d1d000@192.168.203.137@tcp:6/4 lens 488/448 e 0 to 0 dl 1736990967 ref 2 fl Interpret:RMQU/0/0 rc 0/0 job:'directio.0' [ 483.799779] LustreError: 1875:0:(osc_request.c:2397:osc_brw_redo_request()) Skipped 1 previous similar message [ 484.984692] LustreError: 133-1: lustre-OST0001-osc-ffff8800a9d1d000: BAD READ CHECKSUM: from 192.168.203.137@tcp inode [0x200000407:0xc17:0x0] object 0x0:3818 extent [0-4194303], client f5a99216/f5a99216, server de2bf73e, cksum_type 4 [ 487.554398] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 20:29:24 (1736990964) [ 487.962788] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 488.079532] LustreError: 132-0: lustre-OST0000-osc-ffff8800a9d1d000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.203.137@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [ 488.086785] LustreError: Skipped 1 previous similar message [ 490.736602] LustreError: 132-0: lustre-OST0000-osc-ffff8800a9d1d000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.203.137@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [4194304-8388607], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [ 490.745214] LustreError: Skipped 2 previous similar messages [ 497.741791] LustreError: 132-0: lustre-OST0000-osc-ffff8800a9d1d000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.203.137@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [ 497.749269] LustreError: Skipped 3 previous similar messages [ 502.709839] Lustre: *** cfs_fail_loc=409, val=0*** [ 502.711010] Lustre: Skipped 23 previous similar messages [ 502.712534] LustreError: 1876:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88007e3bf200 x1821366560647936/t8589937148(8589937148) o4->lustre-OST0000-osc-ffff8800a9d1d000@192.168.203.137@tcp:6/4 lens 488/448 e 0 to 0 dl 1736990986 ref 2 fl Interpret:RMQU/0/0 rc 0/0 job:'ptlrpcd_00_00.0' [ 502.720416] LustreError: 1876:0:(osc_request.c:2397:osc_brw_redo_request()) Skipped 11 previous similar messages [ 508.718934] LustreError: 132-0: lustre-OST0000-osc-ffff8800a9d1d000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.203.137@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [ 508.724922] LustreError: Skipped 3 previous similar messages [ 526.859156] LustreError: 132-0: lustre-OST0000-osc-ffff8800a9d1d000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.203.137@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060 [ 526.864494] LustreError: Skipped 5 previous similar messages [ 532.904784] LustreError: 1875:0:(osc_request.c:2527:brw_interpret()) lustre-OST0000-osc-ffff8800a9d1d000: too many resent retries for object: 0:3885, rc = -11. [ 533.292119] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 534.763586] Lustre: *** cfs_fail_loc=409, val=0*** [ 534.764663] Lustre: Skipped 27 previous similar messages [ 534.766044] LustreError: 1877:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800ab0eaac0 x1821366560651264/t8589937162(8589937162) o4->lustre-OST0000-osc-ffff8800a9d1d000@192.168.203.137@tcp:6/4 lens 488/448 e 0 to 0 dl 1736991018 ref 2 fl Interpret:RMQU/0/0 rc 0/0 job:'ptlrpcd_01_00.0' [ 534.774165] LustreError: 1877:0:(osc_request.c:2397:osc_brw_redo_request()) Skipped 11 previous similar messages [ 560.808448] LustreError: 132-0: lustre-OST0000-osc-ffff8800a9d1d000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.203.137@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [4194304-8388607], original client csum 7d33b9a0 (type 2), server csum 7d33b99f (type 2), client csum now 7d33b9a0 [ 560.814198] LustreError: Skipped 17 previous similar messages [ 578.829572] LustreError: 1877:0:(osc_request.c:2527:brw_interpret()) lustre-OST0000-osc-ffff8800a9d1d000: too many resent retries for object: 0:3885, rc = -11. [ 578.833102] LustreError: 1877:0:(osc_request.c:2527:brw_interpret()) Skipped 1 previous similar message [ 579.242330] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 600.866029] Lustre: *** cfs_fail_loc=409, val=0*** [ 600.867184] Lustre: Skipped 63 previous similar messages [ 600.868192] LustreError: 1877:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800ab0ee3c0 x1821366560657024/t8589937194(8589937194) o4->lustre-OST0000-osc-ffff8800a9d1d000@192.168.203.137@tcp:6/4 lens 488/448 e 0 to 0 dl 1736991084 ref 2 fl Interpret:RMQU/0/0 rc 0/0 job:'ptlrpcd_01_01.0' [ 600.874401] LustreError: 1877:0:(osc_request.c:2397:osc_brw_redo_request()) Skipped 29 previous similar messages [ 624.912838] LustreError: 132-0: lustre-OST0000-osc-ffff8800a9d1d000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.203.137@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [4194304-8388607], original client csum de2bf73f (type 4), server csum de2bf73e (type 4), client csum now de2bf73f [ 624.920675] LustreError: Skipped 27 previous similar messages [ 624.922334] LustreError: 1877:0:(osc_request.c:2527:brw_interpret()) lustre-OST0000-osc-ffff8800a9d1d000: too many resent retries for object: 0:3885, rc = -11. [ 624.926849] LustreError: 1877:0:(osc_request.c:2527:brw_interpret()) Skipped 1 previous similar message [ 625.335004] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 670.896115] LustreError: 1876:0:(osc_request.c:2527:brw_interpret()) lustre-OST0000-osc-ffff8800a9d1d000: too many resent retries for object: 0:3885, rc = -11. [ 670.904121] LustreError: 1876:0:(osc_request.c:2527:brw_interpret()) Skipped 1 previous similar message [ 671.558656] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 717.050100] LustreError: 1878:0:(osc_request.c:2527:brw_interpret()) lustre-OST0000-osc-ffff8800a9d1d000: too many resent retries for object: 0:3885, rc = -11. [ 717.053136] LustreError: 1878:0:(osc_request.c:2527:brw_interpret()) Skipped 1 previous similar message [ 717.587618] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 733.057894] Lustre: *** cfs_fail_loc=409, val=0*** [ 733.059558] Lustre: Skipped 127 previous similar messages [ 733.061314] LustreError: 1875:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880136a80980 x1821366560668288/t8589937258(8589937258) o4->lustre-OST0000-osc-ffff8800a9d1d000@192.168.203.137@tcp:6/4 lens 488/448 e 0 to 0 dl 1736991216 ref 2 fl Interpret:RMQU/0/0 rc 0/0 job:'ptlrpcd_00_01.0' [ 733.070152] LustreError: 1875:0:(osc_request.c:2397:osc_brw_redo_request()) Skipped 57 previous similar messages [ 757.397426] LustreError: 132-0: lustre-OST0000-osc-ffff8800a9d1d000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 192.168.203.137@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [4194304-8388607], original client csum a30822f0 (type 40), server csum a30822ef (type 40), client csum now a30822f0 [ 757.405584] LustreError: Skipped 63 previous similar messages [ 763.456282] LustreError: 1876:0:(osc_request.c:2527:brw_interpret()) lustre-OST0000-osc-ffff8800a9d1d000: too many resent retries for object: 0:3885, rc = -11. [ 763.461845] LustreError: 1876:0:(osc_request.c:2527:brw_interpret()) Skipped 1 previous similar message [ 763.960587] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 808.523067] LustreError: 1875:0:(osc_request.c:2527:brw_interpret()) lustre-OST0000-osc-ffff8800a9d1d000: too many resent retries for object: 0:3885, rc = -11. [ 808.530618] LustreError: 1875:0:(osc_request.c:2527:brw_interpret()) Skipped 1 previous similar message [ 809.991830] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 811.590582] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 20:34:48 (1736991288) [ 813.817514] LustreError: 133-1: lustre-OST0000-osc-ffff8800a9d1d000: BAD READ CHECKSUM: from 192.168.203.137@tcp inode [0x200000407:0xc19:0x0] object 0x0:3886 extent [0-4194303], client cbad0602/cbad0602, server bab2161a, cksum_type 4 [ 817.106037] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 20:34:53 (1736991293) [ 817.536652] Lustre: Unmounted lustre-client [ 817.537617] Lustre: Skipped 1 previous similar message [ 817.617116] Lustre: Mounted lustre-client [ 817.618207] Lustre: Skipped 1 previous similar message [ 824.378245] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 20:35:00 (1736991300) [ 824.754049] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 825.166163] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 826.957135] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 842.595334] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in IDLE state after 15 sec [ 843.911189] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 844.247829] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in FULL state after 0 sec [ 844.621637] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 845.908133] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 867.827288] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in IDLE state after 21 sec [ 869.775899] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 870.323015] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in FULL state after 0 sec [ 870.961497] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 872.933390] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 892.895802] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in IDLE state after 19 sec [ 894.638220] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 895.101705] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in FULL state after 0 sec [ 895.689503] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 897.516882] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 917.398494] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in IDLE state after 19 sec [ 918.801348] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 919.131491] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in FULL state after 0 sec [ 919.495181] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 920.794050] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 942.597827] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in IDLE state after 21 sec [ 944.190440] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 944.548051] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in FULL state after 0 sec [ 944.907305] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 946.248890] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 967.985277] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in IDLE state after 21 sec [ 969.929220] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 970.329567] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in FULL state after 0 sec [ 970.937803] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 972.710845] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 992.777121] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in IDLE state after 19 sec [ 994.610684] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 995.100090] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in FULL state after 0 sec [ 996.937099] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 997.326432] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 20:37:53 (1736991473) [ 998.946617] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 20:37:55 (1736991475) [ 999.376968] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 999.760605] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1000.237873] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1000.684826] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1002.186521] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1002.590728] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 20:37:59 (1736991479) [ 1005.684008] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 20:38:02 (1736991482) [ 1007.674348] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ================================================================================= 20:38:04 (1736991484) [ 1018.494588] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 20:38:15 (1736991495) [ 1021.272464] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 20:38:17 (1736991497) [ 1023.266990] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 20:38:19 (1736991499) [ 1025.473571] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 20:38:21 (1736991501) [ 1030.140690] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ===================================================================== 20:38:26 (1736991506) [ 1033.477204] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 20:38:30 (1736991510) [ 1055.075595] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 20:38:51 (1736991531) [ 1059.268315] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 20:38:55 (1736991535) [ 1069.563505] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 20:39:05 (1736991545) [ 1106.625635] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 20:39:42 (1736991582) [ 1113.821351] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 20:39:50 (1736991590) [ 1116.280751] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 20:39:52 (1736991592) [ 1117.741084] Lustre: Unmounted lustre-client [ 1117.742291] Lustre: Skipped 4 previous similar messages [ 1117.810743] Lustre: Mounted lustre-client [ 1117.812023] Lustre: Skipped 4 previous similar messages [ 1124.494525] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 20:40:00 (1736991600) [ 1127.211138] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 20:40:03 (1736991603) [ 1129.575362] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 20:40:06 (1736991606) [ 1136.893183] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 20:40:13 (1736991613) [ 1138.624519] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 20:40:15 (1736991615) [ 1141.399994] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 20:40:17 (1736991617) [ 1143.435185] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 20:40:19 (1736991619) [ 1146.583946] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 20:40:23 (1736991623) [ 1150.999937] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 20:40:27 (1736991627) [ 1151.470533] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 1152.038728] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 1152.571218] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 1153.026508] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 1155.242723] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 20:40:31 (1736991631) [ 1156.199552] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 1156.822061] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 1157.393803] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 1157.832447] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 1158.253014] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 1159.892789] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 20:40:36 (1736991636) [ 1161.542161] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 20:40:37 (1736991637) [ 1164.808518] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 20:40:41 (1736991641) [ 1166.619171] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 20:40:43 (1736991643) [ 1168.232362] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 20:40:44 (1736991644) [ 1170.568879] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 20:40:46 (1736991646) [ 1172.725882] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 20:40:49 (1736991649) [ 1175.253876] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 20:40:51 (1736991651) [ 1177.749380] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 20:40:54 (1736991654) [ 1179.936043] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 20:40:56 (1736991656) [ 1179.996589] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800ab273000: operation mds_getxattr to node 192.168.203.137@tcp failed: rc = -95 [ 1181.610380] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 20:40:58 (1736991658) [ 1184.171697] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 20:41:00 (1736991660) [ 1219.004761] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 20:41:35 (1736991695) [ 1230.408054] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 20:41:46 (1736991706) [ 1232.054690] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 20:41:48 (1736991708) [ 1436.944457] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 20:45:13 (1736991913) [ 1443.094337] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 20:45:19 (1736991919) [ 1443.159364] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1443.177664] Lustre: lustre-OST0000-osc-ffff8800ab273000: Connection to lustre-OST0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1443.187364] LustreError: 167-0: lustre-OST0000-osc-ffff8800ab273000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1443.193943] Lustre: lustre-OST0000-osc-ffff8800ab273000: Connection restored to (at 192.168.203.137@tcp) [ 1444.511965] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ab273000.ost_server_uuid 40 [ 1444.864536] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800ab273000.ost_server_uuid in FULL state after 0 sec [ 1446.903736] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 20:45:23 (1736991923) [ 1449.036263] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 20:45:25 (1736991925) [ 1449.540636] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [ 1450.004780] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 20:45:26 (1736991926) [ 1451.657404] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 20:45:28 (1736991928) [ 1453.211511] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 20:45:29 (1736991929) [ 1454.711723] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 20:45:31 (1736991931) [ 1454.776326] LustreError: 27017:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 315 sleeping for 10000ms [ 1464.779729] LustreError: 27017:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 315 awake [ 1466.873258] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 20:45:43 (1736991943) [ 1468.649955] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 20:45:45 (1736991945) [ 1470.277755] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 20:45:46 (1736991946) [ 1473.168751] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 20:45:49 (1736991949) [ 1474.900148] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [ 1475.336104] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 20:45:51 (1736991951) [ 1531.777455] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 20:46:48 (1736992008) [ 1534.658902] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 20:46:51 (1736992011) [ 1536.231793] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 20:46:52 (1736992012) [ 1537.848102] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 20:46:54 (1736992014) [ 1540.178261] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 1540.586973] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 1540.989354] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 20:46:57 (1736992017) [ 1541.048709] Lustre: *** cfs_fail_loc=40a, val=0*** [ 1541.049980] Lustre: Skipped 65 previous similar messages [ 1541.051173] LustreError: 3575:0:(osc_request.c:2750:osc_build_rpc()) prep_req failed: -22 [ 1541.053347] LustreError: 3575:0:(osc_cache.c:2186:osc_check_rpcs()) Write request failed with -22 [ 1542.695716] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 20:46:59 (1736992019) [ 1542.748141] LustreError: 4226:0:(osc_request.c:2750:osc_build_rpc()) prep_req failed: -12 [ 1542.750674] LustreError: 4226:0:(osc_cache.c:2186:osc_check_rpcs()) Write request failed with -12 [ 1544.432177] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 20:47:00 (1736992020) [ 1544.983705] LustreError: 11-0: lustre-OST0001-osc-ffff8800ab273000: operation ost_write to node 192.168.203.137@tcp failed: rc = -5 [ 1544.986315] LustreError: 1877:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800a8ec2600 x1821366575780224/t0(0) o4->lustre-OST0001-osc-ffff8800ab273000@192.168.203.137@tcp:6/4 lens 4584/224 e 0 to 0 dl 1736992028 ref 2 fl Interpret:RMQU/0/0 rc -5/-5 job:'multiop.0' [ 1544.992075] LustreError: 1877:0:(osc_request.c:2397:osc_brw_redo_request()) Skipped 31 previous similar messages [ 1545.998096] LustreError: 11-0: lustre-OST0001-osc-ffff8800ab273000: operation ost_write to node 192.168.203.137@tcp failed: rc = -5 [ 1548.003652] LustreError: 11-0: lustre-OST0001-osc-ffff8800ab273000: operation ost_write to node 192.168.203.137@tcp failed: rc = -5 [ 1554.206184] LustreError: 11-0: lustre-OST0001-osc-ffff8800ab273000: operation ost_write to node 192.168.203.137@tcp failed: rc = -5 [ 1554.208930] LustreError: Skipped 1 previous similar message [ 1554.210063] LustreError: 1877:0:(osc_request.c:2527:brw_interpret()) lustre-OST0001-osc-ffff8800ab273000: too many resent retries for object: 0:6320, rc = -5. [ 1554.213208] LustreError: 1877:0:(osc_request.c:2527:brw_interpret()) Skipped 1 previous similar message [ 1554.215297] Lustre: 1877:0:(llite_lib.c:3590:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.137@tcp:/lustre/fid: [0x20000040a:0xe44:0x0]// may get corrupted (rc -5) [ 1556.078040] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 20:47:12 (1736992032) [ 1564.211600] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 20:47:20 (1736992040) [ 1564.637662] LustreError: 11-0: lustre-OST0001-osc-ffff8800ab273000: operation ost_write to node 192.168.203.137@tcp failed: rc = -14 [ 1564.640627] LustreError: Skipped 3 previous similar messages [ 1566.541297] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 20:47:23 (1736992043) [ 1576.161315] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 20:47:32 (1736992052) [ 1577.866839] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 20:47:34 (1736992054) [ 1579.445980] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 20:47:35 (1736992055) [ 1583.776269] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 20:47:40 (1736992060) [ 1585.312086] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 20:47:41 (1736992061) [ 1586.941086] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 20:47:43 (1736992063) [ 1588.519496] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ========================================================== 20:47:45 (1736992065) [ 1589.922489] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [ 1593.642781] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 20:47:50 (1736992070) [ 1595.810509] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 20:47:52 (1736992072) [ 1597.892743] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 20:47:54 (1736992074) [ 1600.370073] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 20:47:56 (1736992076) [ 1602.517391] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 20:47:59 (1736992079) [ 1611.671656] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 20:48:08 (1736992088) [ 1621.600373] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 20:48:17 (1736992097) [ 1671.361695] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 20:49:07 (1736992147) [ 1671.406949] Lustre: *** cfs_fail_loc=310, val=0*** [ 1672.923452] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 20:49:09 (1736992149) [ 1674.021516] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 0 sec [ 1674.532440] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 1679.440393] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 2 sec [ 1680.317317] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 0 sec [ 1722.985733] Lustre: DEBUG MARKER: ls -l 10000 files without statahead: 24 sec [ 1727.301353] Lustre: DEBUG MARKER: ls -l 10000 files with statahead: 3 sec [ 1727.828737] Lustre: DEBUG MARKER: ls -l done [ 1747.231830] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity/: 19 seconds [ 1747.769052] Lustre: DEBUG MARKER: rm done [ 1750.084050] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 20:50:26 (1736992226) [ 1750.681253] Lustre: DEBUG MARKER: SKIP: sanity test_123ab Test must be statx() syscall supported [ 1751.329288] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 20:50:27 (1736992227) [ 1751.903213] Lustre: DEBUG MARKER: SKIP: sanity test_123ac Test must be statx() syscall supported [ 1752.554109] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 20:50:28 (1736992228) [ 1758.111645] Lustre: *** cfs_fail_loc=803, val=0*** [ 1758.114891] LustreError: 1877:0:(mdc_locks.c:1389:mdc_intent_getattr_async_interpret()) lustre-MDT0000-mdc-ffff8800ab273000: ldlm_cli_enqueue_fini() failed: rc = -110 [ 1762.998426] Lustre: DEBUG MARKER: ls done [ 1767.890143] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 20:50:44 (1736992244) [ 1768.345534] Lustre: Unmounted lustre-client [ 1768.346788] Lustre: Skipped 1 previous similar message [ 1768.461445] Lustre: Mounted lustre-client [ 1768.463182] Lustre: Skipped 1 previous similar message [ 1771.072698] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 20:50:47 (1736992247) [ 1771.716781] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 1780.710289] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ab241000 [ 1781.277172] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ab241000 [ 1781.836252] Lustre: DEBUG MARKER: LRU=1004 [ 1782.275834] Lustre: DEBUG MARKER: LIMIT=46624 [ 1782.866223] Lustre: DEBUG MARKER: LVF=5572500 [ 1783.476179] Lustre: DEBUG MARKER: OLD_LVF=100 [ 1784.071693] Lustre: DEBUG MARKER: Sleep 50 sec [ 1834.747459] Lustre: DEBUG MARKER: Dropped 390 locks in 50s [ 1835.336357] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 1843.484823] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 20:51:59 (1736992319) [ 1872.255817] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 1878.528043] Lustre: DEBUG MARKER: ls -la time: 5 seconds [ 1879.100427] Lustre: DEBUG MARKER: lru_size = 400 [ 1929.011348] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 1935.005774] Lustre: DEBUG MARKER: ls -la time: 5 seconds [ 1935.390449] Lustre: DEBUG MARKER: lru_size = 4006 [ 1935.950285] Lustre: DEBUG MARKER: lru resize performs the same with no lru resize [ 1948.608851] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 20:53:45 (1736992425) [ 1971.745198] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize diasbaled ========================================================== 20:54:08 (1736992448) [ 1994.581354] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 20:54:31 (1736992471) [ 1996.068005] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 20:54:32 (1736992472) [ 1998.142660] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 20:54:34 (1736992474) [ 2000.714423] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 20:54:37 (1736992477) [ 2002.911716] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [ 2008.271924] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 20:54:44 (1736992484) [ 2010.674639] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 20:54:46 (1736992486) [ 2030.100469] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 20:55:06 (1736992506) [ 2032.234792] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 20:55:08 (1736992508) [ 2034.777890] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 20:55:11 (1736992511) [ 2037.187458] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 20:55:13 (1736992513) [ 2037.542020] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [ 2037.947208] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 20:55:14 (1736992514) [ 2047.036809] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 20:55:23 (1736992523) [ 2049.060961] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 20:55:25 (1736992525) [ 2053.524785] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 20:55:29 (1736992529) [ 2055.386567] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 20:55:31 (1736992531) [ 2057.487314] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 20:55:33 (1736992533) [ 2059.636203] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 20:55:36 (1736992536) [ 2062.056718] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 20:55:38 (1736992538) [ 2064.195235] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 20:55:40 (1736992540) [ 2069.257036] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 20:55:45 (1736992545) [ 2073.317474] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 20:55:49 (1736992549) [ 2088.422012] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 20:56:04 (1736992564) [ 2096.637023] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 20:56:13 (1736992573) [ 2100.553180] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 20:56:16 (1736992576) [ 2102.944881] Lustre: 20264:0:(libcfs_string.c:112:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 2102.964027] ------------[ cut here ]------------ [ 2102.964990] WARNING: CPU: 3 PID: 20272 at mm/page_alloc.c:3222 __alloc_pages_nodemask+0x53f/0xca0 [ 2102.966931] Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) crc_t10dif crct10dif_generic rpcsec_gss_krb5 sb_edac edac_core iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel squashfs aesni_intel lrw gf128mul glue_helper ablk_helper cryptd i2c_piix4 i2c_core pcspkr binfmt_misc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_piix libata [ 2102.978296] CPU: 3 PID: 20272 Comm: badarea_io Kdump: loaded Tainted: G OE ------------ 3.10.0-7.9-debug #1 [ 2102.980250] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 2102.981819] Call Trace: [ 2102.982283] [] dump_stack+0x19/0x1b [ 2102.983366] [] __warn+0xd8/0x100 [ 2102.984289] [] warn_slowpath_null+0x1d/0x20 [ 2102.985488] [] __alloc_pages_nodemask+0x53f/0xca0 [ 2102.986783] [] alloc_pages_current+0x98/0x110 [ 2102.987956] [] kmalloc_order+0x1f/0x70 [ 2102.989038] [] kmalloc_order_trace+0x26/0xc0 [ 2102.990343] [] __kmalloc_track_caller+0x30e/0x330 [ 2102.991673] [] ? cfs_trace_dump_debug_buffer_usrstr+0x14/0x60 [libcfs] [ 2102.993291] [] ? __free_pages+0x1d/0x30 [ 2102.994408] [] memdup_user_nul+0x26/0x90 [ 2102.995562] [] cfs_trace_dump_debug_buffer_usrstr+0x14/0x60 [libcfs] [ 2102.997128] [] proc_dump_kernel+0x1e/0x20 [libcfs] [ 2102.998382] [] lnet_debugfs_write+0x43/0x70 [libcfs] [ 2102.999766] [] vfs_write+0xd6/0x230 [ 2103.000900] [] SyS_write+0x7f/0xf0 [ 2103.001924] [] system_call_fastpath+0x1f/0x24 [ 2103.003157] ---[ end trace f25c93a53526c3e5 ]--- [ 2103.216364] Lustre: Unmounted lustre-client [ 2126.953243] Key type lgssc unregistered [ 2127.025165] LNet: 20970:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.029857] LNet: Removed LNI 192.168.203.37@tcp [ 2130.266473] alg: No test for adler32 (adler32-zlib) [ 2131.056523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2131.108881] Lustre: Lustre: Build Version: 2.15.6_6_ga1ea412 [ 2131.156566] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 2131.157891] LNet: Accept secure, port 988 [ 2132.682733] Key type lgssc registered [ 2133.066390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2164.405925] Lustre: Mounted lustre-client [ 2166.566392] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2174.706921] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 20:57:31 (1736992651) [ 2188.755047] Lustre: Unmounted lustre-client [ 2212.037119] Key type lgssc unregistered [ 2212.102052] LNet: 25035:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2213.106191] LNet: Removed LNI 192.168.203.37@tcp [ 2215.378212] alg: No test for adler32 (adler32-zlib) [ 2216.164272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2216.220829] Lustre: Lustre: Build Version: 2.15.6_6_ga1ea412 [ 2216.264588] LNet: Added LNI 192.168.203.37@tcp [8/256/0/180] [ 2216.266123] LNet: Accept secure, port 988 [ 2217.796632] Key type lgssc registered [ 2218.085237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2247.492523] Lustre: Mounted lustre-client [ 2249.907833] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2258.588447] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 20:58:54 (1736992734) [ 2272.517640] Lustre: lustre-OST0000-osc-ffff8800a8f60800: disconnect after 24s idle [ 2453.168786] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 21:02:09 (1736992929) [ 2471.128743] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 21:02:27 (1736992947) [ 2496.694777] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [ 2497.131422] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [ 2497.587738] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 21:02:54 (1736992974) [ 2497.877685] Lustre: lustre-OST0000-osc-ffff8800a8f60800: disconnect after 24s idle [ 2497.880258] Lustre: Skipped 1 previous similar message [ 2504.337926] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 21:03:00 (1736992980) [ 2504.598690] Lustre: Unmounted lustre-client [ 2504.671933] Lustre: Mounted lustre-client [ 2514.040566] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 21:03:10 (1736992990) [ 2524.152380] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 21:03:20 (1736993000) [ 2534.580972] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 21:03:30 (1736993010) [ 2544.581518] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 21:03:41 (1736993021) [ 2548.672262] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 21:03:45 (1736993025) [ 2566.966222] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 21:04:03 (1736993043) [ 2579.546068] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 21:04:15 (1736993055) [ 2588.384497] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 21:04:24 (1736993064) [ 2596.449721] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 21:04:32 (1736993072) [ 2598.249811] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 21:04:34 (1736993074) [ 2599.967090] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 21:04:36 (1736993076) [ 2601.605079] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 21:04:38 (1736993078) [ 2603.220263] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 21:04:39 (1736993079) [ 2605.212393] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 21:04:41 (1736993081) [ 2607.134179] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 21:04:43 (1736993083) [ 2608.783609] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 21:04:45 (1736993085) [ 2610.952205] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 21:04:47 (1736993087) [ 2612.807452] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 21:04:49 (1736993089) [ 2614.943518] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 21:04:51 (1736993091) [ 2732.934801] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 21:06:49 (1736993209) [ 2737.091449] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 21:06:53 (1736993213) [ 2741.126524] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 21:06:57 (1736993217) [ 2745.202676] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 21:07:01 (1736993221) [ 2749.353500] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 21:07:05 (1736993225) [ 2764.759878] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 21:07:21 (1736993241) [ 2785.864277] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 21:07:42 (1736993262) [ 2805.884087] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 21:08:02 (1736993282) [ 2824.090772] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 21:08:20 (1736993300) [ 2826.926625] Lustre: DEBUG MARKER: Turn on read and write cache [ 2828.321055] Lustre: DEBUG MARKER: Write data and read it back. [ 2828.758251] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 2830.009341] Lustre: DEBUG MARKER: cache hits: before: 65581, after: 65584 [ 2830.443236] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 2831.329299] Lustre: DEBUG MARKER: cache hits:: before: 65584, after: 65587 [ 2831.806852] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [ 2832.962724] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 2834.183909] Lustre: DEBUG MARKER: cache hits:: before: 65587, after: 65590 [ 2834.618430] Lustre: DEBUG MARKER: Write data and read it back. [ 2835.019286] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 2836.218943] Lustre: DEBUG MARKER: cache hits:: before: 65590, after: 65593 [ 2836.660891] Lustre: DEBUG MARKER: Turn off read and write cache [ 2837.796675] Lustre: DEBUG MARKER: Write data and read it back [ 2838.214487] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 2839.554288] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65593 [ 2839.990951] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [ 2841.284272] Lustre: DEBUG MARKER: Write data and read it back [ 2841.757546] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 2843.138590] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65593 [ 2843.608342] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 2844.972995] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65596 [ 2847.767103] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 21:08:44 (1736993324) [ 2855.224812] Lustre: lustre-MDT0000-mdc-ffff8800ab066000: Connection to lustre-MDT0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2860.234152] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail [ 2860.242472] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf941611282 to 0x725a8bf941675233 [ 2860.247105] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 192.168.203.137@tcp) [ 2860.250753] LustreError: 25351:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800a9dcf200 x1821368851003840/t12884906963(12884906963) o101->lustre-MDT0000-mdc-ffff8800ab066000@192.168.203.137@tcp:12/10 lens 576/600 e 0 to 0 dl 1736993381 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'cmp.0' [ 2862.377242] Lustre: lustre-MDT0000-mdc-ffff8800ab066000: Connection restored to (at 192.168.203.137@tcp) [ 2867.683466] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 21:09:04 (1736993344) [ 2874.092100] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 21:09:10 (1736993350) [ 2880.916433] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 21:09:17 (1736993357) [ 2886.649107] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 21:09:23 (1736993363) [ 2892.689361] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 21:09:29 (1736993369) [ 2896.678406] Lustre: DEBUG MARKER: 1736993373: creating first files [ 2922.071586] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 21:09:58 (1736993398) [ 2937.507315] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 21:10:13 (1736993413) [ 2965.399509] Lustre: lustre-MDT0000-mdc-ffff8800ab066000: Connection to lustre-MDT0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2965.399510] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail [ 2965.407984] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf941675233 to 0x725a8bf9416762cc [ 2965.415155] Lustre: Skipped 1 previous similar message [ 2965.417350] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 192.168.203.137@tcp) [ 2965.425912] LustreError: 25351:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800a9dcf200 x1821368851003840/t12884906963(12884906963) o101->lustre-MDT0000-mdc-ffff8800ab066000@192.168.203.137@tcp:12/10 lens 576/600 e 0 to 0 dl 1736993485 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'cmp.0' [ 2965.435192] LustreError: 25351:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 6 previous similar messages [ 2967.605854] Lustre: lustre-MDT0001-mdc-ffff8800ab066000: Connection restored to 192.168.203.137@tcp (at 192.168.203.137@tcp) [ 2973.945744] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 21:10:50 (1736993450) [ 2985.857504] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 21:11:02 (1736993462) [ 2986.006297] Lustre: Mounted lustre-client [ 2987.700181] Lustre: Unmounted lustre-client [ 2989.330094] Lustre: Mounted lustre-client [ 2991.391876] Lustre: Unmounted lustre-client [ 2991.859254] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 21:11:08 (1736993468) [ 3004.292689] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 21:11:20 (1736993480) [ 3011.413917] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 21:11:27 (1736993487) [ 3021.881304] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 21:11:38 (1736993498) [ 3303.598014] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 21:16:20 (1736993780) [ 3312.707374] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 21:16:29 (1736993789) [ 3314.854453] Lustre: lustre-MDT0000-mdc-ffff88007e1a9000: Connection to lustre-MDT0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3319.863088] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail [ 3319.867441] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf9416762cc to 0x725a8bf941aad993 [ 3319.871613] LustreError: 25351:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800a86063c0 x1821368854221504/t21474836531(21474836531) o101->lustre-MDT0000-mdc-ffff88007e1a9000@192.168.203.137@tcp:12/10 lens 576/600 e 0 to 0 dl 1736993839 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'lfs.0' [ 3319.871698] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 192.168.203.137@tcp) [ 3319.871700] Lustre: Skipped 1 previous similar message [ 3319.883125] LustreError: 25351:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 10 previous similar messages [ 3325.064782] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 21:16:41 (1736993801) [ 3327.942964] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records anaconda-ks.cfg stress.sh time ========================================================== 21:16:44 (1736993804) [ 3339.354487] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 21:16:55 (1736993815) [ 3346.719055] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 21:17:03 (1736993823) [ 3353.882027] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 21:17:10 (1736993830) [ 3359.113393] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 21:17:15 (1736993835) [ 3360.634632] LustreError: 3337:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 140c sleeping for 5000ms [ 3362.736623] LustreError: 3337:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 3365.812623] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 21:17:22 (1736993842) [ 3367.628652] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 21:17:24 (1736993844) [ 3369.407827] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 21:17:25 (1736993845) [ 3380.700739] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 21:17:37 (1736993857) [ 3389.974879] Lustre: lustre-OST0000-osc-ffff88007e1a9000: Connection to lustre-OST0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3393.012869] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 21:17:49 (1736993869) [ 3419.930214] Lustre: lustre-OST0000-osc-ffff88007e1a9000: Connection to lustre-OST0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3419.940388] Lustre: lustre-OST0000-osc-ffff88007e1a9000: Connection restored to 192.168.203.137@tcp (at 192.168.203.137@tcp) [ 3419.943785] Lustre: Skipped 1 previous similar message [ 3420.080036] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 21:18:16 (1736993896) [ 3430.038740] Lustre: lustre-OST0000-osc-ffff88007e1a9000: Connection to lustre-OST0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3433.719830] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 21:18:30 (1736993910) [ 3455.078805] Lustre: lustre-OST0000-osc-ffff88007e1a9000: Connection to lustre-OST0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3457.905160] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 21:18:54 (1736993934) [ 3458.046289] Lustre: lustre-OST0000-osc-ffff88007e1a9000: Connection restored to 192.168.203.137@tcp (at 192.168.203.137@tcp) [ 3458.048664] Lustre: Skipped 1 previous similar message [ 3472.367951] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 21:19:08 (1736993948) [ 3484.389051] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 21:19:20 (1736993960) [ 3484.775811] Lustre: DEBUG MARKER: creating a 10 Mb file [ 3485.202762] Lustre: DEBUG MARKER: starting reads [ 3485.597583] Lustre: DEBUG MARKER: truncating the file [ 3486.040969] Lustre: DEBUG MARKER: killing dd [ 3486.435801] Lustre: DEBUG MARKER: removing the temporary file [ 3488.099570] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log =============================================================================== 21:19:24 (1736993964) [ 3488.131731] Lustre: debug daemon will attempt to start writing to /tmp/f170.sanity_log_good (512000kB max) [ 3488.146232] Lustre: shutting down debug daemon thread... [ 3488.153872] Lustre: debug daemon will attempt to start writing to /tmp/f170.sanity_log_good (512000kB max) [ 3488.164116] Lustre: shutting down debug daemon thread... [ 3489.787265] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 21:19:26 (1736993966) [ 3489.822925] Lustre: *** cfs_fail_loc=50e, val=0*** [ 3489.823101] LustreError: dumping log to /tmp/lustre-log.1736993966.10605 [ 3489.826647] Lustre: Skipped 1 previous similar message [ 3489.832054] LustreError: 15633:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50e sleeping for 3000ms [ 3492.834619] LustreError: 15633:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50e awake [ 3492.838037] LustreError: dumping log to /tmp/lustre-log.1736993969.15633 [ 3494.427327] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 21:19:30 (1736993970) [ 3494.822699] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [ 3495.285677] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 21:19:31 (1736993971) [ 3500.464317] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 21:19:36 (1736993976) [ 3506.060438] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 21:19:42 (1736993982) [ 3520.930487] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ========================================================================== 21:19:57 (1736993997) [ 3529.936392] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 21:20:06 (1736994006) [ 3532.247062] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 21:20:08 (1736994008) [ 3534.823022] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 21:20:11 (1736994011) [ 3536.941404] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 21:20:13 (1736994013) [ 3544.447472] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 21:20:20 (1736994020) [ 3549.399320] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 21:20:25 (1736994025) [ 3553.648242] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 21:20:30 (1736994030) [ 3555.270565] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 21:20:31 (1736994031) [ 3558.055203] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 21:20:34 (1736994034) [ 3563.881529] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 21:20:40 (1736994040) [ 3565.844378] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 21:20:42 (1736994042) [ 3567.504431] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 21:20:44 (1736994044) [ 3583.465230] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 21:20:59 (1736994059) [ 3585.211944] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 21:21:01 (1736994061) [ 3587.448760] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 21:21:03 (1736994063) [ 3589.194760] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 21:21:05 (1736994065) [ 3591.503846] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 21:21:07 (1736994067) [ 3593.487696] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 21:21:09 (1736994069) [ 3595.668197] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 21:21:11 (1736994071) [ 3597.884520] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 21:21:14 (1736994074) [ 3600.045476] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 21:21:16 (1736994076) [ 3606.053691] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [ 3606.414304] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.15342 [ 3607.130246] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [ 3607.673153] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.3707 [ 3608.642852] Lustre: DEBUG MARKER: Test: lfs mkdir -i 1 /mnt/lustre/d205a.sanity.remote [ 3609.177041] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.21854 [ 3610.090755] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [ 3610.518593] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.9355 [ 3611.165034] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [ 3611.588866] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.23240 [ 3612.245311] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [ 3612.630527] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.13112 [ 3613.268692] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3613.648589] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.24291 [ 3614.509904] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [ 3615.056792] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.13792 [ 3615.842553] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [ 3616.229450] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.14893 [ 3616.937472] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [ 3617.329100] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.3329 [ 3618.348272] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [ 3618.889920] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.24748 [ 3619.623260] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [ 3620.072588] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.17141 [ 3623.918069] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3624.377760] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg337-client.v [ 3625.254311] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3625.703883] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg337-client.E [ 3626.357096] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3626.736359] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.oleg337-client.v [ 3630.493002] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 21:21:46 (1736994106) [ 3633.417444] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 21:21:49 (1736994109) [ 3635.123995] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 21:21:51 (1736994111) [ 3635.172437] Lustre: *** cfs_fail_loc=1403, val=0*** [ 3635.174384] LustreError: 12140:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240001b72:0x1c4a:0x0]: rc = -5 [ 3635.178363] LustreError: 12140:0:(llite_lib.c:3195:ll_prep_inode()) new_inode -fatal: rc -5 [ 3636.785578] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 21:21:53 (1736994113) [ 3638.452008] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 21:21:54 (1736994114) [ 3640.109325] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 21:21:56 (1736994116) [ 3650.391581] Lustre: lustre-MDT0000-mdc-ffff88007e1a9000: Connection to lustre-MDT0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3650.397993] Lustre: Skipped 2 previous similar messages [ 3657.397670] Lustre: 25352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736994127/real 1736994127] req@ffff8800a7b20e40 x1821368878844352/t0(0) o400->MGC192.168.203.137@tcp@192.168.203.137@tcp:26/25 lens 224/224 e 0 to 1 dl 1736994134 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3657.404075] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail [ 3662.430601] LustreError: 25351:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800a7b27200 x1821368878843840/t25769826229(25769826229) o101->lustre-MDT0000-mdc-ffff88007e1a9000@192.168.203.137@tcp:12/10 lens 576/600 e 0 to 0 dl 1736994181 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'' [ 3663.441773] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf941aad993 to 0x725a8bf941bf7452 [ 3663.445137] Lustre: MGC192.168.203.137@tcp: Connection restored to 192.168.203.137@tcp (at 192.168.203.137@tcp) [ 3663.447661] Lustre: Skipped 2 previous similar messages [ 3667.314141] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3667.690954] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3679.461622] Lustre: 25353:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736994149/real 1736994149] req@ffff8800a7b24c00 x1821368878848256/t0(0) o400->MGC192.168.203.137@tcp@192.168.203.137@tcp:26/25 lens 224/224 e 0 to 1 dl 1736994156 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3679.467743] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail [ 3685.487337] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf941bf7452 to 0x725a8bf941bf765f [ 3690.419195] LustreError: 25351:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88009d94da40 x1821368878847936/t30064771075(30064771075) o101->lustre-MDT0000-mdc-ffff88007e1a9000@192.168.203.137@tcp:12/10 lens 584/600 e 0 to 0 dl 1736994209 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'' [ 3690.427304] LustreError: 25351:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 1 previous similar message [ 3695.144941] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3695.509684] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3698.412209] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 21:22:54 (1736994174) [ 3703.637823] bash (16384): drop_caches: 3 [ 3705.036770] bash (16384): drop_caches: 3 [ 3706.918547] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 21:23:03 (1736994183) [ 3710.544785] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 21:23:07 (1736994187) [ 3712.458978] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 21:23:08 (1736994188) [ 3712.513088] LustreError: 25354:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 40f sleeping for 10000ms [ 3722.557653] LustreError: 25354:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 40f awake [ 3724.162258] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 21:23:20 (1736994200) [ 3728.266993] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 21:23:24 (1736994204) [ 3729.975133] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 21:23:26 (1736994206) [ 3735.458377] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') ========================================================== 21:23:31 (1736994211) [ 3737.539821] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 21:23:34 (1736994214) [ 3737.897811] Lustre: DEBUG MARKER: creating a 10 Mb file [ 3738.335663] Lustre: DEBUG MARKER: starting reads [ 3738.728515] Lustre: DEBUG MARKER: truncating the file [ 3739.139646] Lustre: DEBUG MARKER: killing dd [ 3739.515526] Lustre: DEBUG MARKER: removing the temporary file [ 3741.166572] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 21:23:37 (1736994217) [ 3741.198971] Lustre: *** cfs_fail_loc=411, val=0*** [ 3742.894136] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 21:23:39 (1736994219) [ 3754.591899] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 21:23:51 (1736994231) [ 3756.640084] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 21:23:53 (1736994233) [ 3758.548338] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 21:23:55 (1736994235) [ 3760.411394] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 21:23:56 (1736994236) [ 3762.337759] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 21:23:58 (1736994238) [ 3762.390525] Lustre: *** cfs_fail_loc=508, val=2147483648*** [ 3762.393228] LustreError: 25344:0:(events.c:213:client_bulk_callback()) event type 1, status -5, desc ffff8800acaf6800 [ 3762.397858] Lustre: 25354:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1736994239/real 1736994239] req@ffff88012d0e0e40 x1821368879128768/t0(0) o4->lustre-OST0001-osc-ffff88007e1a9000@192.168.203.137@tcp:6/4 lens 488/448 e 0 to 1 dl 1736994246 ref 2 fl Rpc:eXQr/0/ffffffff rc 0/-1 job:'' [ 3762.411147] Lustre: lustre-OST0001-osc-ffff88007e1a9000: Connection to lustre-OST0001 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3762.416506] Lustre: Skipped 1 previous similar message [ 3765.481906] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 21:24:01 (1736994241) [ 3773.014895] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 21:24:09 (1736994249) [ 3784.486690] Lustre: 30288:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736994256/real 1736994256] req@ffff88012d0e17c0 x1821368879135040/t0(0) o35->lustre-MDT0000-mdc-ffff88007e1a9000@192.168.203.137@tcp:23/10 lens 392/624 e 0 to 1 dl 1736994261 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3784.504078] Lustre: lustre-MDT0000-mdc-ffff88007e1a9000: Connection restored to 192.168.203.137@tcp (at 192.168.203.137@tcp) [ 3784.508235] Lustre: Skipped 3 previous similar messages [ 3791.675616] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 21:24:28 (1736994268) [ 3812.541670] Lustre: 25354:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736994269/real 1736994269] req@ffff8800764fdf00 x1821368879140672/t0(0) o3->lustre-OST0000-osc-ffff88007e1a9000@192.168.203.137@tcp:6/4 lens 488/440 e 0 to 1 dl 1736994289 ref 2 fl Bulk:RXMQU/0/0 rc 0/0 job:'' [ 3812.550498] LustreError: 25354:0:(client.c:2170:ptlrpc_check_set()) @@@ bulk transfer failed 0/1048576/0 req@ffff8800764fdf00 x1821368879140672/t0(0) o3->lustre-OST0000-osc-ffff88007e1a9000@192.168.203.137@tcp:6/4 lens 488/440 e 0 to 1 dl 1736994289 ref 2 fl Bulk:ReXMQU/0/0 rc 0/0 job:'' [ 3812.558497] LustreError: 25354:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800764fdf00 x1821368879140672/t0(0) o3->lustre-OST0000-osc-ffff88007e1a9000@192.168.203.137@tcp:6/4 lens 488/440 e 0 to 1 dl 1736994289 ref 2 fl Interpret:ReXMQU/0/0 rc -5/0 job:'' [ 3816.056687] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [ 3816.529704] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [ 3816.978855] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 21:24:53 (1736994293) [ 3818.741676] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 21:24:55 (1736994295) [ 3820.658632] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 21:24:57 (1736994297) [ 3820.784092] Lustre: Mounted lustre-client [ 3822.060780] Lustre: Unmounted lustre-client [ 3822.455051] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 21:24:58 (1736994298) [ 3824.099496] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 21:25:00 (1736994300) [ 3824.706270] Lustre: *** cfs_fail_loc=1002, val=0*** [ 3894.932104] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 21:26:11 (1736994371) [ 3895.712349] Lustre: *** cfs_fail_loc=1002, val=0*** [ 3976.056511] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail [ 3976.056942] Lustre: lustre-MDT0000-mdc-ffff88007e1a9000: Connection to lustre-MDT0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3976.056943] Lustre: Skipped 2 previous similar messages [ 3976.074729] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf941bf765f to 0x725a8bf941d67bf6 [ 3976.083913] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 192.168.203.137@tcp) [ 4001.485741] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 21:27:57 (1736994477) [ 4002.523447] Lustre: *** cfs_fail_loc=1002, val=0*** [ 4170.649389] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 21:30:47 (1736994647) [ 4172.821575] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 21:30:49 (1736994649) [ 4175.229928] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 21:30:51 (1736994651) [ 4186.064914] LustreError: cowardly refusing to write 4285 bytes in a page [ 4186.067078] LustreError: 8578:0:(namei.c:1176:ll_atomic_open()) VFS Op:name=lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll [ 4186.096845] LustreError: cowardly refusing to write 4286 bytes in a page [ 4186.098762] LustreError: 8579:0:(namei.c:1176:ll_atomic_open()) VFS Op:name=lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll [ 4189.001904] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 21:31:05 (1736994665) [ 4189.595397] LustreError: 9353:0:(file.c:259:ll_close_inode_openhandle()) lustre-clilmv-ffff88007e1a9000: inode [0x200003ab1:0x2fec:0x0] mdc close failed: rc = -116 [ 4192.533374] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [ 4193.117660] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 21:31:09 (1736994669) [ 4195.651615] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 21:31:12 (1736994672) [ 4198.320367] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 21:31:14 (1736994674) [ 4200.325730] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 21:31:16 (1736994676) [ 4202.407458] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 21:31:18 (1736994678) [ 4204.640332] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 21:31:21 (1736994681) [ 4206.825941] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 21:31:23 (1736994683) [ 4207.306399] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [ 4207.883322] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 21:31:24 (1736994684) [ 4236.032381] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 21:31:52 (1736994712) [ 4236.427178] bash (15066): drop_caches: 3 [ 4236.673458] bash (15066): drop_caches: 3 [ 4238.803210] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 21:31:55 (1736994715) [ 4240.935074] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 21:31:57 (1736994717) [ 4254.095232] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 21:32:10 (1736994730) [ 4266.802952] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 21:32:23 (1736994743) [ 4277.493726] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 21:32:33 (1736994753) [ 4279.477931] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 21:32:35 (1736994755) [ 4282.426372] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 21:32:38 (1736994758) [ 4284.303972] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 21:32:40 (1736994760) [ 4284.755122] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [ 4285.275294] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 21:32:41 (1736994761) [ 4285.707306] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [ 4286.192633] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 21:32:42 (1736994762) [ 4288.084994] Lustre: DEBUG MARKER: == sanity test 230y: unlink dir with bad hash type ======= 21:32:44 (1736994764) [ 4293.840687] Lustre: DEBUG MARKER: == sanity test 230z: resume dir migration with bad hash type ========================================================== 21:32:50 (1736994770) [ 4302.941250] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 21:32:59 (1736994779) [ 4306.023273] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 21:33:02 (1736994782) [ 4311.612293] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 21:33:08 (1736994788) [ 4311.911619] LustreError: 11-0: lustre-OST0000-osc-ffff88007e1a9000: operation ldlm_enqueue to node 192.168.203.137@tcp failed: rc = -12 [ 4312.410636] Lustre: Unmounted lustre-client [ 4312.543211] Lustre: Mounted lustre-client [ 4316.952892] Lustre: lustre-OST0000-osc-ffff8800ab067000: Connection to lustre-OST0000 (at 192.168.203.137@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4317.220662] Lustre: lustre-OST0000-osc-ffff8800ab067000: Connection restored to 192.168.203.137@tcp (at 192.168.203.137@tcp) [ 4317.225138] Lustre: Skipped 1 previous similar message [ 4318.998537] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 21:33:15 (1736994795) [ 4319.785962] Lustre: Unmounted lustre-client [ 4319.871523] Lustre: Mounted lustre-client [ 4326.580025] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 21:33:23 (1736994803) [ 4328.364550] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 21:33:24 (1736994804) [ 4330.246381] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 21:33:26 (1736994806) [ 4330.360114] Lustre: *** cfs_fail_loc=1405, val=0*** [ 4332.130413] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 21:33:28 (1736994808) [ 4335.993702] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 21:33:32 (1736994812) [ 4337.870283] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 21:33:34 (1736994814) [ 4339.735048] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 21:33:36 (1736994816) [ 4359.678126] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 21:33:56 (1736994836) [ 4366.884129] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 21:34:03 (1736994843) [ 4372.825188] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 21:34:09 (1736994849) [ 4373.144634] Lustre: Unmounted lustre-client [ 4373.544046] Lustre: Mounted lustre-client [ 4378.353094] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 21:34:14 (1736994854) [ 4399.263530] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 21:34:35 (1736994875) [ 4408.334778] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 21:34:44 (1736994884) [ 4408.712462] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800b39c6000: operation mds_readpage to node 192.168.203.137@tcp failed: rc = -12 [ 4410.797207] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 21:34:47 (1736994887) [ 4411.966937] Lustre: 12580:0:(file.c:2680:ll_get_grouplock()) group lock already existed with gid 97486 [ 4411.969766] Lustre: 12580:0:(file.c:2752:ll_put_grouplock()) no group lock held [ 4411.972121] Lustre: 12580:0:(file.c:2665:ll_get_grouplock()) group id for group lock must not be 0 [ 4411.976146] Lustre: 12580:0:(file.c:2760:ll_put_grouplock()) group lock 4294967286 doesn't match current id 3543 [ 4425.134850] Lustre: 12580:0:(file.c:2752:ll_put_grouplock()) no group lock held [ 4425.142455] Lustre: 12580:0:(file.c:2665:ll_get_grouplock()) group id for group lock must not be 0 [ 4427.166761] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 21:35:03 (1736994903) [ 4439.160927] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 21:35:15 (1736994915) [ 4441.185720] Lustre: DEBUG MARKER: == sanity test 245: check mdc connection flag/data: multiple modify RPCs ========================================================== 21:35:17 (1736994917) [ 4443.115946] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 21:35:19 (1736994919) [ 4443.218793] Lustre: Mounted lustre-client [ 4443.328629] Lustre: Unmounted lustre-client [ 4445.641484] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 21:35:22 (1736994922) [ 4445.753855] LustreError: 15961:0:(llite_lib.c:646:client_common_fill_super()) cannot mds_connect: rc = -2 [ 4445.778769] LustreError: 15961:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4447.718825] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 21:35:24 (1736994924) [ 4450.165406] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 21:35:26 (1736994926) [ 4452.726867] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 21:35:29 (1736994929) [ 4452.844758] Lustre: Mounted lustre-client [ 4452.846196] Lustre: Skipped 5 previous similar messages [ 4452.945218] Lustre: Unmounted lustre-client [ 4452.946817] Lustre: Skipped 6 previous similar messages [ 4453.002057] LustreError: 11-0: lustre-MDT0000-mdc-ffff880075a28000: operation mds_get_root to node 192.168.203.137@tcp failed: rc = -22 [ 4453.005598] LustreError: 18066:0:(llite_lib.c:646:client_common_fill_super()) cannot mds_connect: rc = -22 [ 4453.028505] LustreError: 18066:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4454.980267] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 21:35:31 (1736994931) [ 4456.265460] LustreError: 11-0: lustre-MDT0000-mdc-ffff880075a2e000: operation mds_get_root to node 192.168.203.137@tcp failed: rc = -66 [ 4456.269672] LustreError: 18843:0:(llite_lib.c:646:client_common_fill_super()) cannot mds_connect: rc = -66 [ 4456.291048] LustreError: 18843:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -66 [ 4461.140682] Lustre: DEBUG MARKER: == sanity test 247g: mount striped directory as fileset caches ROOT lookup lock ========================================================== 21:35:37 (1736994937) [ 4461.567530] Lustre: DEBUG MARKER: SKIP: sanity test_247g needs >= 4 MDTs [ 4462.060068] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 21:35:38 (1736994938) [ 4469.420709] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 21:35:45 (1736994945) [ 4477.523587] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 21:35:54 (1736994954) [ 4479.288030] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 21:35:55 (1736994955) [ 4481.251156] Lustre: DEBUG MARKER: == sanity test 251: Handling short read and write correctly ========================================================== 21:35:57 (1736994957) [ 4481.299042] Lustre: *** cfs_fail_loc=1407, val=0*** [ 4483.377832] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 21:35:59 (1736994959) [ 4486.544937] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 21:36:02 (1736994962) [ 4583.794749] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 21:37:40 (1736995060) [ 4589.673199] Lustre: DEBUG MARKER: SKIP: sanity test_255a skipping excluded test 255a (base 255) [ 4590.196401] Lustre: DEBUG MARKER: SKIP: sanity test_255b skipping excluded test 255b (base 255) [ 4590.751484] Lustre: DEBUG MARKER: SKIP: sanity test_255c skipping excluded test 255c (base 255) [ 4591.300973] Lustre: DEBUG MARKER: SKIP: sanity test_256 skipping excluded test 256 [ 4591.779978] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 21:37:48 (1736995068) [ 4592.236300] LustreError: 11-0: lustre-MDT0001-mdc-ffff8800b39c6000: operation ldlm_enqueue to node 192.168.203.137@tcp failed: rc = -14 [ 4609.797231] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 21:38:06 (1736995086) [ 4611.579821] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 21:38:08 (1736995088) [ 4613.307805] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 21:38:09 (1736995089) [ 4633.872672] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 21:38:30 (1736995110) [ 4633.897631] Lustre: *** cfs_fail_loc=806, val=0*** [ 4633.899068] Lustre: 31778:0:(mdc_request.c:959:mdc_close()) lustre-MDT0000-mdc-ffff8800b39c6000: close of FID [0x200004283:0x39:0x0] failed, file reference will be dropped when this client unmounts or is evicted [ 4633.904681] LustreError: 31778:0:(file.c:259:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b39c6000: inode [0x200004283:0x39:0x0] mdc close failed: rc = -12 [ 4635.656568] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 21:38:32 (1736995112) [ 4639.176405] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 21:38:35 (1736995115) [ 4641.199595] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 21:38:37 (1736995117) [ 4643.235698] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 21:38:39 (1736995119) [ 4645.252805] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 21:38:41 (1736995121) [ 4647.360992] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 21:38:43 (1736995123) [ 4652.146106] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 21:38:48 (1736995128) [ 4659.664030] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 21:38:56 (1736995136) [ 4662.406015] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 21:38:58 (1736995138) [ 4664.220721] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 21:39:00 (1736995140) [ 4666.000682] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 21:39:02 (1736995142) [ 4667.764632] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 21:39:04 (1736995144) [ 4669.516333] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 21:39:06 (1736995146) [ 4690.104692] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 21:39:26 (1736995166) [ 4691.992104] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 21:39:28 (1736995168) [ 4693.790268] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 21:39:30 (1736995170) [ 4694.842296] Lustre: *** cfs_fail_loc=314, val=0*** [ 4696.450138] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 21:39:32 (1736995172) [ 4698.258986] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 21:39:34 (1736995174) [ 4700.844809] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 21:39:37 (1736995177) [ 4703.198366] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 21:39:39 (1736995179) [ 4705.618450] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 21:39:42 (1736995182) [ 4707.550013] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 21:39:44 (1736995184) [ 4709.448031] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 21:39:45 (1736995185) [ 4711.224293] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 21:39:47 (1736995187) [ 4715.143531] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 21:39:51 (1736995191) [ 4715.367851] LustreError: 13275:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 31f sleeping for 4000ms [ 4717.070580] LustreError: 13275:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 4718.402291] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 21:39:54 (1736995194) [ 4737.270393] Lustre: 25351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736995207/real 1736995207] req@ffff8800a8f93dc0 x1821368904641984/t0(0) o400->lustre-OST0000-osc-ffff8800b39c6000@192.168.203.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1736995214 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 4745.699699] Lustre: 25351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736995215/real 1736995215] req@ffff8800a8f92ac0 x1821368904643328/t0(0) o400->lustre-OST0000-osc-ffff8800b39c6000@192.168.203.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1736995222 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 4754.254668] Lustre: 25351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736995224/real 1736995224] req@ffff8800a8efb440 x1821368904644544/t0(0) o400->lustre-OST0000-osc-ffff8800b39c6000@192.168.203.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1736995231 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 4762.708612] Lustre: 25351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736995232/real 1736995232] req@ffff8800a8efa140 x1821368904645760/t0(0) o400->lustre-OST0000-osc-ffff8800b39c6000@192.168.203.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1736995239 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 4780.717704] Lustre: 25351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736995239/real 1736995239] req@ffff8800a8efc280 x1821368904646528/t0(0) o400->lustre-OST0000-osc-ffff8800b39c6000@192.168.203.137@tcp:28/4 lens 224/224 e 1 to 1 dl 1736995257 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 4800.090625] Lustre: 25351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736995269/real 1736995269] req@ffff8800a8ef8e40 x1821368904649920/t0(0) o400->lustre-OST0000-osc-ffff8800b39c6000@192.168.203.137@tcp:28/4 lens 224/224 e 0 to 1 dl 1736995276 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 4803.220416] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 21:41:19 (1736995279) [ 4805.129990] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 21:41:21 (1736995281) [ 4809.766997] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail [ 4809.772192] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf941fa30ec to 0x725a8bf941ffa470 [ 4811.182718] LustreError: 25351:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800acbe2140 x1821368904651840/t38654728816(38654728816) o101->lustre-MDT0000-mdc-ffff8800b39c6000@192.168.203.137@tcp:12/10 lens 576/600 e 0 to 0 dl 1736995297 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dd.0' [ 4811.190065] LustreError: 25351:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 2 previous similar messages [ 4822.588692] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 21:41:39 (1736995299) [ 4822.840661] Lustre: Unmounted lustre-client [ 4822.841674] Lustre: Skipped 13 previous similar messages [ 4828.150775] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 192.168.203.137@tcp) was lost; in progress operations using this service will fail [ 4828.154648] LustreError: 15c-8: MGC192.168.203.137@tcp: Confguration from log lustre-client failed from MGS -5. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4828.155417] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf941ffaaa5 to 0x725a8bf941ffab00 [ 4828.172263] LustreError: 23988:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -5 [ 4845.491514] Lustre: Mounted lustre-client [ 4845.493042] Lustre: Skipped 11 previous similar messages [ 4847.269133] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 21:42:03 (1736995323) [ 4849.420395] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 21:42:05 (1736995325) [ 4871.693804] Lustre: DEBUG MARKER: == sanity test 300c: chown [ 4890.291480] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 21:42:46 (1736995366) [ 4892.313367] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ========================================================== 21:42:48 (1736995368) [ 4894.072478] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ========================================================== 21:42:50 (1736995370) [ 4895.759923] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ========================================================== 21:42:52 (1736995372) [ 4898.346790] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory ========================================================== 21:42:54 (1736995374) [ 4901.145567] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory ========================================================== 21:42:57 (1736995377) [ 4901.526188] Lustre: Unmounted lustre-client [ 4901.527518] Lustre: Skipped 1 previous similar message [ 4901.603990] Lustre: Mounted lustre-client [ 4901.744088] Lustre: *** cfs_fail_loc=1901, val=0*** [ 4903.950403] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ============ 21:43:00 (1736995380) [ 4905.827503] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ======== 21:43:02 (1736995382) [ 4907.940093] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ========================================================== 21:43:04 (1736995384) [ 4909.882033] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS ========================================================== 21:43:06 (1736995386) [ 4910.316922] Lustre: DEBUG MARKER: SKIP: sanity test_300m Only for single MDT [ 4910.819248] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA ========================================================== 21:43:07 (1736995387) [ 4915.031448] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [ 4915.489705] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================================== 21:43:11 (1736995391) [ 4917.541018] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ========================================================== 21:43:14 (1736995394) [ 4919.278326] Lustre: DEBUG MARKER: == sanity test 300r: test -1 striped directory =========== 21:43:15 (1736995395) [ 4921.008764] Lustre: DEBUG MARKER: == sanity test 300s: test lfs mkdir -c without -i ======== 21:43:17 (1736995397) [ 4922.794882] Lustre: DEBUG MARKER: == sanity test 300t: test max_mdt_stripecount ============ 21:43:19 (1736995399) [ 4925.276917] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ============= 21:43:21 (1736995401) [ 4927.111659] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open ========================================================== 21:43:23 (1736995403) [ 4928.985749] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ========================================================== 21:43:25 (1736995405) [ 4929.403857] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [ 4929.890466] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ========================================================== 21:43:26 (1736995406) [ 4948.196602] Lustre: DEBUG MARKER: == sanity test 312: make sure ZFS adjusts its block size by write pattern ========================================================== 21:43:44 (1736995424) [ 4948.691367] Lustre: DEBUG MARKER: SKIP: sanity test_312 the test only applies to zfs [ 4949.186732] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ========================================================== 21:43:45 (1736995425) [ 4951.687608] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ========================================================== 21:43:48 (1736995428) [ 4961.263837] LustreError: 25354:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff880137131300 x1821368907194880/t68719480331(68719480331) o4->lustre-OST0000-osc-ffff88008e15a800@192.168.203.137@tcp:6/4 lens 648/448 e 0 to 0 dl 1736995445 ref 3 fl Interpret:RQU/4/0 rc -5/-5 job:'diff.0' [ 4962.272459] LustreError: 25354:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff88012a484c00 x1821368907194944/t68719480332(68719480332) o4->lustre-OST0000-osc-ffff88008e15a800@192.168.203.137@tcp:6/4 lens 648/448 e 0 to 0 dl 1736995446 ref 3 fl Interpret:RQU/4/0 rc -5/-5 job:'ptlrpcd_01_00.0' [ 4964.265216] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ============= 21:44:00 (1736995440) [ 4968.195440] Lustre: DEBUG MARKER: == sanity test 316: lfs migrate of file with large_xattr enabled ========================================================== 21:44:04 (1736995444) [ 4970.351590] Lustre: DEBUG MARKER: == sanity test 317: Verify blocks get correctly update after truncate ========================================================== 21:44:06 (1736995446) [ 4972.414161] Lustre: DEBUG MARKER: == sanity test 318: Verify async readahead tunables ====== 21:44:08 (1736995448) [ 4972.498566] LustreError: 12582:0:(lproc_llite.c:1183:read_ahead_async_file_threshold_mb_store()) lustre: can't set read_ahead_async_file_threshold_mb=30 > max_read_readahead_per_file_mb=29 [ 4974.240749] Lustre: DEBUG MARKER: == sanity test 319: lost lease lock on migrate error ===== 21:44:10 (1736995450) [ 4974.315666] LustreError: 13219:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms [ 4979.318668] LustreError: 13219:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake [ 4979.332788] LustreError: 13219:0:(file.c:259:ll_close_inode_openhandle()) lustre-clilmv-ffff88008e15a800: inode [0x200005223:0x40a:0x0] mdc close failed: rc = -116 [ 4981.130645] Lustre: DEBUG MARKER: == sanity test 350: force NID mismatch path to be exercised ========================================================== 21:44:17 (1736995457) [ 5014.055422] Lustre: DEBUG MARKER: == sanity test 398a: direct IO should cancel lock otherwise lockless ========================================================== 21:44:50 (1736995490) [ 5015.890314] Lustre: DEBUG MARKER: == sanity test 398b: DIO and buffer IO race ============== 21:44:52 (1736995492) [ 5020.869033] Lustre: DEBUG MARKER: == sanity test 398c: run fio to test AIO ================= 21:44:57 (1736995497) [ 5031.453184] Lustre: DEBUG MARKER: == sanity test 398d: run aiocp to verify block size > stripe size ========================================================== 21:45:07 (1736995507) [ 5035.116852] Lustre: DEBUG MARKER: == sanity test 398e: O_Direct open cleared by fcntl doesn't cause hang ========================================================== 21:45:11 (1736995511) [ 5036.792425] Lustre: DEBUG MARKER: == sanity test 398f: verify aio handles ll_direct_rw_pages errors correctly ========================================================== 21:45:13 (1736995513) [ 5039.035066] Lustre: DEBUG MARKER: == sanity test 398g: verify parallel dio async RPC submission ========================================================== 21:45:15 (1736995515) [ 5060.226316] Lustre: DEBUG MARKER: == sanity test 398h: verify correctness of read [ 5063.724975] Lustre: DEBUG MARKER: == sanity test 398i: verify parallel dio handles ll_direct_rw_pages errors correctly ========================================================== 21:45:40 (1736995540) [ 5064.618129] Lustre: *** cfs_fail_loc=1418, val=0*** [ 5066.385054] Lustre: DEBUG MARKER: == sanity test 398j: test parallel dio where stripe size > rpc_size ========================================================== 21:45:42 (1736995542) [ 5070.183739] Lustre: DEBUG MARKER: == sanity test 398k: test enospc on first stripe ========= 21:45:46 (1736995546) [ 5085.892601] Lustre: DEBUG MARKER: SKIP: sanity test_398k 7023424 > 600000 skipping out-of-space test on OST0 [ 5086.380700] Lustre: DEBUG MARKER: == sanity test 398l: test enospc on intermediate stripe/RPC ========================================================== 21:46:02 (1736995562) [ 5091.250513] Lustre: DEBUG MARKER: SKIP: sanity test_398l 7003512 > 600000 skipping out-of-space test on OST0 [ 5091.733340] Lustre: DEBUG MARKER: == sanity test 398m: test RPC failures with parallel dio ========================================================== 21:46:08 (1736995568) [ 5092.084185] LustreError: 11-0: lustre-OST0000-osc-ffff88008e15a800: operation ost_write to node 192.168.203.137@tcp failed: rc = -5 [ 5092.088971] LustreError: 25352:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800acb32f80 x1821368914012544/t0(0) o4->lustre-OST0000-osc-ffff88008e15a800@192.168.203.137@tcp:6/4 lens 488/224 e 0 to 0 dl 1736995613 ref 2 fl Interpret:ReMQU/0/0 rc -5/-5 job:'dd.0' [ 5095.100866] LustreError: 25353:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800acb350c0 x1821368914013824/t0(0) o4->lustre-OST0000-osc-ffff88008e15a800@192.168.203.137@tcp:6/4 lens 488/224 e 0 to 0 dl 1736995616 ref 2 fl Interpret:ReMQU/0/0 rc -5/-5 job:'ptlrpcd_00_00.0' [ 5095.106928] LustreError: 25353:0:(osc_request.c:2397:osc_brw_redo_request()) Skipped 7 previous similar messages [ 5102.114699] LustreError: 25353:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800acb36880 x1821368914015040/t0(0) o4->lustre-OST0000-osc-ffff88008e15a800@192.168.203.137@tcp:6/4 lens 488/224 e 0 to 0 dl 1736995623 ref 2 fl Interpret:ReMQU/0/0 rc -5/-5 job:'ptlrpcd_00_00.0' [ 5102.121968] LustreError: 25353:0:(osc_request.c:2397:osc_brw_redo_request()) Skipped 6 previous similar messages [ 5113.128027] LustreError: 11-0: lustre-OST0000-osc-ffff88008e15a800: operation ost_write to node 192.168.203.137@tcp failed: rc = -5 [ 5113.131023] LustreError: Skipped 23 previous similar messages [ 5113.132259] LustreError: 25353:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800acb31c80 x1821368914015744/t0(0) o4->lustre-OST0000-osc-ffff88008e15a800@192.168.203.137@tcp:6/4 lens 488/224 e 0 to 0 dl 1736995634 ref 2 fl Interpret:ReMQU/0/0 rc -5/-5 job:'ptlrpcd_00_00.0' [ 5113.138291] LustreError: 25353:0:(osc_request.c:2397:osc_brw_redo_request()) Skipped 7 previous similar messages [ 5137.145994] LustreError: 25352:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800acb350c0 x1821368914018112/t0(0) o4->lustre-OST0000-osc-ffff88008e15a800@192.168.203.137@tcp:6/4 lens 488/224 e 0 to 0 dl 1736995659 ref 2 fl Interpret:ReMQU/0/0 rc -5/-5 job:'ptlrpcd_00_01.0' [ 5137.153787] LustreError: 25352:0:(osc_request.c:2397:osc_brw_redo_request()) Skipped 11 previous similar messages [ 5147.157926] LustreError: 11-0: lustre-OST0000-osc-ffff88008e15a800: operation ost_write to node 192.168.203.137@tcp failed: rc = -5 [ 5147.161680] LustreError: Skipped 15 previous similar messages [ 5147.163017] LustreError: 25353:0:(osc_request.c:2527:brw_interpret()) lustre-OST0000-osc-ffff88008e15a800: too many resent retries for object: 0:31257, rc = -5. [ 5169.426061] LustreError: 25353:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800acb36880 x1821368914033024/t0(0) o3->lustre-OST0000-osc-ffff88008e15a800@192.168.203.137@tcp:6/4 lens 488/440 e 0 to 0 dl 1736995691 ref 2 fl Interpret:ReMQU/0/0 rc -5/-5 job:'ptlrpcd_00_00.0' [ 5169.434415] LustreError: 25353:0:(osc_request.c:2397:osc_brw_redo_request()) Skipped 27 previous similar messages [ 5203.442525] LustreError: 25353:0:(osc_request.c:2527:brw_interpret()) lustre-OST0000-osc-ffff88008e15a800: too many resent retries for object: 0:31257, rc = -5. [ 5203.447716] LustreError: 25353:0:(osc_request.c:2527:brw_interpret()) Skipped 3 previous similar messages [ 5214.201043] LustreError: 11-0: lustre-OST0001-osc-ffff88008e15a800: operation ost_write to node 192.168.203.137@tcp failed: rc = -5 [ 5214.204600] LustreError: Skipped 63 previous similar messages [ 5240.212117] LustreError: 25353:0:(osc_request.c:2397:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800acb350c0 x1821368914043456/t0(0) o4->lustre-OST0001-osc-ffff88008e15a800@192.168.203.137@tcp:6/4 lens 488/224 e 0 to 0 dl 1736995761 ref 2 fl Interpret:ReMQU/0/0 rc -5/-5 job:'ptlrpcd_00_00.0' [ 5240.219406] LustreError: 25353:0:(osc_request.c:2397:osc_brw_redo_request()) Skipped 46 previous similar messages [ 5259.224261] LustreError: 25353:0:(osc_request.c:2527:brw_interpret()) lustre-OST0001-osc-ffff88008e15a800: too many resent retries for object: 0:29669, rc = -5. [ 5259.228201] LustreError: 25353:0:(osc_request.c:2527:brw_interpret()) Skipped 3 previous similar messages [ 5315.756137] LustreError: 25355:0:(osc_request.c:2527:brw_interpret()) lustre-OST0001-osc-ffff88008e15a800: too many resent retries for object: 0:29669, rc = -5. [ 5315.760586] LustreError: 25355:0:(osc_request.c:2527:brw_interpret()) Skipped 3 previous similar messages [ 5318.210570] Lustre: DEBUG MARKER: == sanity test 398n: test append with parallel DIO ======= 21:49:54 (1736995794) [ 5321.583595] Lustre: DEBUG MARKER: == sanity test 399a: fake write should not be slower than normal write ========================================================== 21:49:58 (1736995798) [ 5348.235999] Lustre: DEBUG MARKER: == sanity test 399b: fake read should not be slower than normal read ========================================================== 21:50:24 (1736995824) [ 5354.975178] Lustre: DEBUG MARKER: SKIP: sanity test_400a skipping excluded test 400a [ 5355.405072] Lustre: DEBUG MARKER: == sanity test 400b: packaged headers can be compiled ==== 21:50:31 (1736995831) [ 5357.045256] Lustre: DEBUG MARKER: == sanity test 401a: Verify if 'lctl list_param -R' can list parameters recursively ========================================================== 21:50:33 (1736995833) [ 5358.812034] Lustre: DEBUG MARKER: == sanity test 401b: Verify 'lctl get_param' set_param' continue after error ========================================================== 21:50:35 (1736995835) [ 5360.494118] Lustre: DEBUG MARKER: == sanity test 401c: Verify 'lctl set_param' without value fails in either format. ========================================================== 21:50:36 (1736995836) [ 5362.105338] Lustre: DEBUG MARKER: == sanity test 401d: Verify 'lctl set_param' accepts values containing '=' ========================================================== 21:50:38 (1736995838) [ 5363.839973] Lustre: DEBUG MARKER: == sanity test 401e: verify 'lctl get_param' works with NID in parameter ========================================================== 21:50:40 (1736995840) [ 5365.553033] Lustre: DEBUG MARKER: == sanity test 402: Return ENOENT to lod_generate_and_set_lovea ========================================================== 21:50:42 (1736995842) [ 5367.578989] Lustre: DEBUG MARKER: == sanity test 403: i_nlink should not drop to zero due to aliasing ========================================================== 21:50:44 (1736995844) [ 5367.673158] sysctl (3444): drop_caches: 2 [ 5369.367104] Lustre: DEBUG MARKER: == sanity test 404: validate manual {de}activated works properly for OSPs ========================================================== 21:50:45 (1736995845) [ 5375.228404] Lustre: DEBUG MARKER: == sanity test 405: Various layout swap lock tests ======= 21:50:51 (1736995851) [ 5378.634127] Lustre: DEBUG MARKER: SKIP: sanity test_405 layout swap does not support DOM files so far [ 5379.137545] Lustre: DEBUG MARKER: == sanity test 406: DNE support fs default striping ====== 21:50:55 (1736995855) [ 5394.572609] Lustre: DEBUG MARKER: SKIP: sanity test_407 skipping ALWAYS excluded test 407 [ 5395.094002] Lustre: DEBUG MARKER: == sanity test 408: drop_caches should not hang due to page leaks ========================================================== 21:51:11 (1736995871) [ 5395.141814] Lustre: *** cfs_fail_loc=40a, val=0*** [ 5395.143039] LustreError: 25354:0:(osc_request.c:2750:osc_build_rpc()) prep_req failed: -22 [ 5395.145637] LustreError: 25354:0:(osc_cache.c:2210:osc_check_rpcs()) Read request failed with -22 [ 5396.358107] bash (6163): drop_caches: 2 [ 5398.972720] Lustre: DEBUG MARKER: == sanity test 409: Large amount of cross-MDTs hard links on the same file ========================================================== 21:51:15 (1736995875) [ 5417.180480] Lustre: DEBUG MARKER: == sanity test 410: Test inode number returned from kernel thread ========================================================== 21:51:33 (1736995893) [ 5417.248304] lustre_kinode_22399: CONFIG_X86_X32 is not set [ 5417.252002] lustre_kinode_22399: inode is 144115540850379563 [ 5417.253857] lustre_kinode_22399: inode is 144115540850379563 [ 5417.255709] lustre_kinode_22399: inode numbers are identical: 144115540850379563 [ 5419.060427] Lustre: DEBUG MARKER: == sanity test 411: Slab allocation error with cgroup does not LBUG ========================================================== 21:51:35 (1736995895) [ 5421.490524] Lustre: DEBUG MARKER: == sanity test 412: mkdir on specific MDTs =============== 21:51:37 (1736995897) [ 5423.663761] Lustre: DEBUG MARKER: == sanity test 413a: QoS mkdir with 'lfs mkdir -i -1' ==== 21:51:40 (1736995900) [ 5592.248055] Lustre: DEBUG MARKER: == sanity test 413b: QoS mkdir under dir whose default LMV starting MDT offset is -1 ========================================================== 21:54:28 (1736996068) [ 5632.813441] Lustre: DEBUG MARKER: == sanity test 413c: mkdir with default LMV max inherit rr ========================================================== 21:55:08 (1736996108) [ 5677.643296] Lustre: DEBUG MARKER: == sanity test 413d: inherit ROOT default LMV ============ 21:55:53 (1736996153) [ 5687.373156] Lustre: DEBUG MARKER: == sanity test 413e: check default max-inherit value ===== 21:56:03 (1736996163) [ 5691.314111] Lustre: DEBUG MARKER: == sanity test 413f: lfs getdirstripe -D list ROOT default LMV if it's not set on dir ========================================================== 21:56:07 (1736996167) [ 5695.333823] Lustre: DEBUG MARKER: == sanity test 413z: 413 test cleanup ==================== 21:56:11 (1736996171) [ 5709.431486] Lustre: DEBUG MARKER: == sanity test 414: simulate ENOMEM in ptlrpc_register_bulk() ========================================================== 21:56:25 (1736996185) [ 5709.597782] Lustre: *** cfs_fail_loc=521, val=0*** [ 5709.603863] LustreError: 25352:0:(niobuf.c:390:ptlrpc_register_bulk()) lustre-OST0000-osc-ffff88008e15a800: LNetMEAttach failed x1821368920241601/1: rc = -12 [ 5713.173236] Lustre: DEBUG MARKER: == sanity test 415: lock revoke is not missing =========== 21:56:29 (1736996189) [ 5736.070937] Lustre: DEBUG MARKER: == sanity test 416: transaction start failure won't cause system hung ========================================================== 21:56:52 (1736996212) [ 5740.094782] Lustre: DEBUG MARKER: == sanity test 417: disable remote dir, striped dir and dir migration ========================================================== 21:56:56 (1736996216) [ 5750.529629] Lustre: DEBUG MARKER: == sanity test 418: df and lfs df outputs match ========== 21:57:06 (1736996226) [ 5787.307890] Lustre: DEBUG MARKER: sanity test_418: @@@@@@ FAIL: df and lfs df inodes output mismatch: df -i: 192.168.203.137@tcp:/lustre 532178 18548 513630 4% /mnt/lustre, lfs df -i: filesystem_summary: 527147 13517 513630 3% /mnt/lustre