[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 309159428 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.459032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.462073] pid_max: default: 32768 minimum: 301 [ 0.463952] Security Framework initialized [ 0.465634] SELinux: Initializing. [ 0.467099] SELinux: Starting in permissive mode [ 0.468579] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.472771] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.476053] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.478531] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.481185] Initializing cgroup subsys memory [ 0.482503] Initializing cgroup subsys devices [ 0.484225] Initializing cgroup subsys freezer [ 0.485640] Initializing cgroup subsys net_cls [ 0.487455] Initializing cgroup subsys blkio [ 0.489360] Initializing cgroup subsys perf_event [ 0.491940] Initializing cgroup subsys hugetlb [ 0.493575] Initializing cgroup subsys pids [ 0.494943] Initializing cgroup subsys net_prio [ 0.496822] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.499659] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.501798] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.503560] tlb_flushall_shift: 6 [ 0.505260] FEATURE SPEC_CTRL Present [ 0.506822] FEATURE IBPB_SUPPORT Present [ 0.508359] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.511044] Spectre V2 : Vulnerable [ 0.512505] Speculative Store Bypass: Vulnerable [ 0.515405] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.522730] ACPI: Core revision 20130517 [ 0.526117] ACPI: All ACPI Tables successfully acquired [ 0.528184] ftrace: allocating 30294 entries in 119 pages [ 0.580651] Enabling x2apic [ 0.581431] Enabled x2apic [ 0.582557] Switched APIC routing to physical x2apic. [ 0.586341] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.588918] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.592392] TSC deadline timer enabled [ 0.592591] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.595781] ... version: 2 [ 0.596899] ... bit width: 48 [ 0.598381] ... generic registers: 4 [ 0.599716] ... value mask: 0000ffffffffffff [ 0.601273] ... max period: 00007fffffffffff [ 0.602793] ... fixed-purpose events: 3 [ 0.604188] ... event mask: 000000070000000f [ 0.606091] KVM setup paravirtual spinlock [ 0.611302] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.614707] KVM setup async PF for cpu 1 [ 0.616183] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.618978] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.609293] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.623828] KVM setup async PF for cpu 2 [ 0.623837] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.627355] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.631659] Brought up 4 CPUs [ 0.631664] KVM setup async PF for cpu 3 [ 0.631672] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.635712] smpboot: Max logical packages: 1 [ 0.637032] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.641371] devtmpfs: initialized [ 0.643209] x86/mm: Memory block size: 128MB [ 0.648116] EVM: security.selinux [ 0.649319] EVM: security.ima [ 0.650390] EVM: security.capability [ 0.654086] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.656948] NET: Registered protocol family 16 [ 0.658441] cpuidle: using governor haltpoll [ 0.660308] ACPI: bus type PCI registered [ 0.661609] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.664161] PCI: Using configuration type 1 for base access [ 0.666090] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.676129] ACPI: Added _OSI(Module Device) [ 0.677940] ACPI: Added _OSI(Processor Device) [ 0.679651] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.681219] ACPI: Added _OSI(Processor Aggregator Device) [ 0.683007] ACPI: Added _OSI(Linux-Dell-Video) [ 0.685302] ACPI: EC: Look up EC in DSDT [ 0.687943] ACPI: Interpreter enabled [ 0.689314] ACPI: (supports S0 S3 S4 S5) [ 0.690615] ACPI: Using IOAPIC for interrupt routing [ 0.692423] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.695913] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.703106] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.705262] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.707727] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.709811] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.713653] acpiphp: Slot [2] registered [ 0.714995] acpiphp: Slot [3] registered [ 0.716427] acpiphp: Slot [4] registered [ 0.717991] acpiphp: Slot [5] registered [ 0.719553] acpiphp: Slot [6] registered [ 0.721157] acpiphp: Slot [7] registered [ 0.722632] acpiphp: Slot [8] registered [ 0.723962] acpiphp: Slot [9] registered [ 0.725489] acpiphp: Slot [10] registered [ 0.726991] acpiphp: Slot [11] registered [ 0.728089] acpiphp: Slot [12] registered [ 0.729598] acpiphp: Slot [13] registered [ 0.731233] acpiphp: Slot [14] registered [ 0.732489] acpiphp: Slot [15] registered [ 0.733868] acpiphp: Slot [16] registered [ 0.734753] acpiphp: Slot [17] registered [ 0.735644] acpiphp: Slot [18] registered [ 0.736989] acpiphp: Slot [19] registered [ 0.738587] acpiphp: Slot [20] registered [ 0.740131] acpiphp: Slot [21] registered [ 0.741686] acpiphp: Slot [22] registered [ 0.743090] acpiphp: Slot [23] registered [ 0.744404] acpiphp: Slot [24] registered [ 0.745667] acpiphp: Slot [25] registered [ 0.746770] acpiphp: Slot [26] registered [ 0.748165] acpiphp: Slot [27] registered [ 0.749384] acpiphp: Slot [28] registered [ 0.750835] acpiphp: Slot [29] registered [ 0.751855] acpiphp: Slot [30] registered [ 0.753188] acpiphp: Slot [31] registered [ 0.754413] PCI host bridge to bus 0000:00 [ 0.755851] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.758221] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.760313] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.762576] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.765026] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.767555] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.769188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.769819] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.770655] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.783075] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.785297] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.787430] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.788945] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.790805] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.795538] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.796220] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.798857] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.801892] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.804149] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.815599] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.817205] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.818286] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.823561] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.833342] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.854427] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.861776] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.867240] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.873110] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.886218] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.889864] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.892850] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.895672] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.907212] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.915111] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.918920] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.923674] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.941895] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.949129] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.951969] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.954641] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.971352] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.976747] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.980686] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.987917] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.002509] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 1.012334] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.016595] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.020559] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.024414] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.027190] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.029784] vgaarb: loaded [ 1.030925] SCSI subsystem initialized [ 1.032295] ACPI: bus type USB registered [ 1.033589] usbcore: registered new interface driver usbfs [ 1.035401] usbcore: registered new interface driver hub [ 1.037219] usbcore: registered new device driver usb [ 1.039435] PCI: Using ACPI for IRQ routing [ 1.040877] PCI: pci_cache_line_size set to 64 bytes [ 1.041239] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.041247] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.041250] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.042207] NetLabel: Initializing [ 1.043412] NetLabel: domain hash size = 128 [ 1.044855] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.046705] NetLabel: unlabeled traffic allowed by default [ 1.048490] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.049959] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.055606] amd_nb: Cannot enumerate AMD northbridges [ 1.057322] Switched to clocksource kvm-clock [ 1.076627] pnp: PnP ACPI init [ 1.077944] ACPI: bus type PNP registered [ 1.079374] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.079466] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.079512] pnp 00:02: [dma 2] [ 1.079544] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.079623] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.079699] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.079790] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.080152] pnp: PnP ACPI: found 6 devices [ 1.081407] ACPI: bus type PNP unregistered [ 1.098464] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.098471] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.098474] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.098476] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.098478] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.098551] NET: Registered protocol family 2 [ 1.100262] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.102963] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.106145] TCP: Hash tables configured (established 32768 bind 32768) [ 1.108463] TCP: reno registered [ 1.109616] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.111622] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.113802] NET: Registered protocol family 1 [ 1.116020] RPC: Registered named UNIX socket transport module. [ 1.118427] RPC: Registered udp transport module. [ 1.119704] RPC: Registered tcp transport module. [ 1.121246] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.123450] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.125347] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.127183] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.129337] PCI: CLS 0 bytes, default 64 [ 1.130850] Unpacking initramfs... [ 2.539477] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.544898] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.548723] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.554553] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.559315] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.562677] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.565903] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.572050] cryptomgr_test (52) used greatest stack depth: 14480 bytes left [ 2.572657] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.572712] Initialise system trusted keyring [ 2.605132] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.607492] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.615164] zpool: loaded [ 2.616049] zbud: loaded [ 2.617187] VFS: Disk quotas dquot_6.6.0 [ 2.618637] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.621491] NFS: Registering the id_resolver key type [ 2.623316] Key type id_resolver registered [ 2.624740] Key type id_legacy registered [ 2.625971] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.628573] Key type big_key registered [ 2.629984] SELinux: Registering netfilter hooks [ 2.632945] cryptomgr_test (58) used greatest stack depth: 14048 bytes left [ 2.634736] cryptomgr_test (59) used greatest stack depth: 13968 bytes left [ 2.637438] NET: Registered protocol family 38 [ 2.639268] Key type asymmetric registered [ 2.640879] Asymmetric key parser 'x509' registered [ 2.642865] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.646093] io scheduler noop registered [ 2.647580] io scheduler deadline registered (default) [ 2.649543] io scheduler cfq registered [ 2.651027] io scheduler mq-deadline registered [ 2.652788] io scheduler kyber registered [ 2.656478] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.658704] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.661427] intel_idle: does not run on family 6 model 62 [ 2.661555] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.664363] ACPI: Power Button [PWRF] [ 2.666259] GHES: HEST is not enabled! [ 2.725438] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.788142] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.916866] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.981851] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 3.107439] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.142030] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 3.172419] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 3.175831] Non-volatile memory driver v1.3 [ 3.178166] Linux agpgart interface v0.103 [ 3.180339] crash memory driver: version 1.1 [ 3.182614] nbd: registered device at major 43 [ 3.189526] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 3.189570] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 3.189600] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 3.189631] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 3.189661] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 3.196698] virtio_blk virtio1: [vda] 115448 512-byte logical blocks (59.1 MB/56.3 MiB) [ 3.204274] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 3.204337] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 3.204388] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 3.204430] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 3.204471] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 3.212023] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.218281] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 3.218362] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 3.218416] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 3.218461] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 3.218501] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 3.224353] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.231396] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 3.231453] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 3.231494] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 3.231530] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 3.231567] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 3.238568] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.245465] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 3.245512] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 3.245546] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 3.245577] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 3.245608] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 3.251918] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.258503] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 3.258556] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 3.258600] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 3.258641] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 3.258690] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 3.264913] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.274110] rdac: device handler registered [ 3.275917] hp_sw: device handler registered [ 3.277593] emc: device handler registered [ 3.279158] libphy: Fixed MDIO Bus: probed [ 3.287538] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.289668] ehci-pci: EHCI PCI platform driver [ 3.291243] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.293235] ohci-pci: OHCI PCI platform driver [ 3.294980] uhci_hcd: USB Universal Host Controller Interface driver [ 3.297605] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.301332] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.302661] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.304607] mousedev: PS/2 mouse device common for all mice [ 3.307365] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.308245] rtc_cmos 00:05: RTC can wake from S4 [ 3.308867] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.309438] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.314833] hidraw: raw HID events driver (C) Jiri Kosina [ 3.315093] usbcore: registered new interface driver usbhid [ 3.315094] usbhid: USB HID core driver [ 3.315183] drop_monitor: Initializing network drop monitor service [ 3.315244] Netfilter messages via NETLINK v0.30. [ 3.315333] TCP: cubic registered [ 3.315341] Initializing XFRM netlink socket [ 3.316266] NET: Registered protocol family 10 [ 3.317034] NET: Registered protocol family 17 [ 3.317086] Key type dns_resolver registered [ 3.317614] mce: Using 10 MCE banks [ 3.318433] Loading compiled-in X.509 certificates [ 3.319503] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.319733] registered taskstats version 1 [ 3.322484] modprobe (71) used greatest stack depth: 13456 bytes left [ 3.325645] Key type trusted registered [ 3.327658] modprobe (77) used greatest stack depth: 13360 bytes left [ 3.330339] Key type encrypted registered [ 3.330413] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.332267] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.334132] rtc_cmos 00:05: setting system clock to 2025-05-02 00:01:59 UTC (1746144119) [ 3.366431] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.369335] Write protecting the kernel read-only data: 12288k [ 3.372571] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.375366] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.386028] random: systemd: uninitialized urandom read (16 bytes read) [ 3.390201] random: systemd: uninitialized urandom read (16 bytes read) [ 3.392240] random: systemd: uninitialized urandom read (16 bytes read) [ 3.396297] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.402329] systemd[1]: Detected virtualization kvm. [ 3.404236] systemd[1]: Detected architecture x86-64. [ 3.406077] systemd[1]: Running in initial RAM disk. [ 3.413109] systemd[1]: No hostname configured. [ 3.415316] systemd[1]: Set hostname to . [ 3.417133] random: systemd: uninitialized urandom read (16 bytes read) [ 3.419611] systemd[1]: Initializing machine ID from random generator. [ 3.471839] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 3.475695] random: systemd: uninitialized urandom read (16 bytes read) [ 3.478146] random: systemd: uninitialized urandom read (16 bytes read) [ 3.480552] random: systemd: uninitialized urandom read (16 bytes read) [ 3.482995] random: systemd: uninitialized urandom read (16 bytes read) [ 3.486713] random: systemd: uninitialized urandom read (16 bytes read) [ 3.489167] random: systemd: uninitialized urandom read (16 bytes read) [ 3.500152] systemd[1]: Created slice Root Slice. [ 3.504073] systemd[1]: Listening on Journal Socket. [ 3.511074] systemd[1]: Reached target Local File Systems. [ 3.515975] systemd[1]: Reached target Timers. [ 3.519857] systemd[1]: Listening on udev Control Socket. [ 3.524299] systemd[1]: Listening on udev Kernel Socket. [ 3.528925] systemd[1]: Reached target Sockets. [ 3.532886] systemd[1]: Created slice System Slice. [ 3.538193] systemd[1]: Starting Setup Virtual Console... [ 3.544237] systemd[1]: Starting Journal Service... [ 3.547488] systemd[1]: Reached target Slices. [ 3.553826] systemd[1]: Starting Load Kernel Modules... [ 3.559251] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.567915] systemd[1]: Starting dracut cmdline hook... [ 3.572258] tsc: Refined TSC clocksource calibration: 2399.978 MHz [ 3.575542] systemd[1]: Reached target Swap. [ 3.579901] systemd[1]: Started Setup Virtual Console. [ 3.587787] systemd[1]: Started Load Kernel Modules. [ 3.592787] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 3.599067] systemd[1]: Started Journal Service. [ 3.824054] random: fast init done [ 4.167614] libata version 3.00 loaded. [ 4.170384] ata_piix 0000:00:01.1: version 2.13 [ 4.177576] scsi host0: ata_piix [ 4.184395] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 4.186127] scsi host1: ata_piix [ 4.186243] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.186245] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 4.393216] ip (322) used greatest stack depth: 13080 bytes left [ 4.450627] ip (345) used greatest stack depth: 12336 bytes left [ 7.526758] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 7.964471] systemd-journald[101]: Received SIGTERM from PID 1 (systemd). [ 8.211299] SELinux: Disabled at runtime. [ 8.212767] SELinux: Unregistering netfilter hooks [ 8.277408] ip_tables: (C) 2000-2006 Netfilter Core Team [ 8.280657] systemd[1]: Inserted module 'ip_tables' [ 8.740060] systemd-journald[568]: Received request to flush runtime journal from PID 1 [ 8.880502] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 8.919774] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 8.952770] cryptd: max_cpu_qlen set to 1000 [ 8.971497] blkid (610) used greatest stack depth: 12160 bytes left [ 8.983115] AVX version of gcm_enc/dec engaged. [ 8.984836] AES CTR mode by8 optimization enabled [ 9.003895] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 9.029728] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 9.046429] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 9.049623] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 9.160425] EDAC MC: Ver: 3.0.0 [ 9.166798] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 9.166810] EDAC sbridge: Ver: 1.1.2 [ 11.306053] mount.nfs (772) used greatest stack depth: 10704 bytes left [ 28.522211] libcfs: loading out-of-tree module taints kernel. [ 28.524502] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 28.563713] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 34.076705] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 34.311471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 34.317456] alg: No test for adler32 (adler32-zlib) [ 35.235082] Lustre: Lustre: Build Version: 2.16.54_72_gbbb30b7 [ 35.448228] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 35.450849] LNet: Accept secure, port 988 [ 37.011474] Key type lgssc registered [ 37.403725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 37.942448] icp: module license 'CDDL' taints kernel. [ 37.944387] Disabling lock debugging due to kernel taint [ 39.596068] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 42.118745] vdc: vdc1 vdc9 [ 45.602277] vde: vde1 vde9 [ 49.495190] vdf: vdf1 vdf9 [ 56.261090] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 59.545314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 60.715855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 60.855228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 60.899599] Lustre: lustre-MDT0000: new disk, initializing [ 61.153799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 61.204167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 61.347819] mount.lustre (7022) used greatest stack depth: 9968 bytes left [ 62.790619] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 65.562038] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 68.023130] Lustre: 8349:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 68.049120] Lustre: lustre-OST0000: new disk, initializing [ 68.054010] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 68.057633] Lustre: Skipped 1 previous similar message [ 68.179280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 70.711618] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 71.763283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 71.766845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 71.838693] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 74.934918] random: crng init done [ 76.197175] Lustre: 9689:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 76.212766] Lustre: lustre-OST0001: new disk, initializing [ 76.216941] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 76.250855] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 78.569016] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 79.891350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 79.895212] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 79.955178] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 85.063607] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 88.958906] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 96.236114] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing check_logdir /tmp/testlogs/ [ 98.210473] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing yml_node [ 100.448244] Lustre: DEBUG MARKER: Client: 2.16.54.72 [ 101.843779] Lustre: DEBUG MARKER: MDS: 2.16.54.72 [ 103.145246] Lustre: DEBUG MARKER: OSS: 2.16.54.72 [ 104.068934] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Thu May 1 20:03:39 EDT 2025 [ 109.955322] hrtimer: interrupt took 3039734 ns [ 112.259398] Lustre: DEBUG MARKER: - need mds1 <= 2.14.55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 112.883993] Lustre: DEBUG MARKER: - need mds1 < v2_14_55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 113.487516] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42c 42b 118c 118d 407 119i 851 411b 130b 130c 130d 130e 130f 130g 312 [ 114.166337] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 51b [ 114.876791] Lustre: DEBUG MARKER: === sanity: start setup 20:03:50 (1746144230) === [ 116.872828] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing check_config_client /mnt/lustre [ 125.764643] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 127.415669] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 128.588845] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 130.156452] Lustre: DEBUG MARKER: === sanity: finish setup 20:04:05 (1746144245) === [ 133.743730] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 20:04:08 (1746144248) [ 135.568827] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 136.422194] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 20:04:11 (1746144251) [ 140.224782] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 20:04:15 (1746144255) [ 257.115864] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 20:06:12 (1746144372) [ 259.965782] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 20:06:15 (1746144375) [ 260.734352] Lustre: *** cfs_fail_loc=15b, val=0*** [ 260.739409] Lustre: *** cfs_fail_loc=15b, val=0*** [ 263.804508] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 20:06:19 (1746144379) [ 266.801341] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 20:06:22 (1746144382) [ 267.257632] Lustre: *** cfs_fail_loc=19a, val=0*** [ 267.768901] Lustre: *** cfs_fail_loc=19a, val=0*** [ 269.095237] Lustre: *** cfs_fail_loc=19a, val=0*** [ 269.096872] Lustre: Skipped 2 previous similar messages [ 271.358532] Lustre: *** cfs_fail_loc=19a, val=0*** [ 271.360512] Lustre: Skipped 4 previous similar messages [ 275.396696] Lustre: *** cfs_fail_loc=19a, val=0*** [ 275.398209] Lustre: Skipped 7 previous similar messages [ 283.738742] Lustre: *** cfs_fail_loc=19a, val=0*** [ 283.744175] Lustre: Skipped 18 previous similar messages [ 300.004797] Lustre: *** cfs_fail_loc=19a, val=0*** [ 300.006413] Lustre: Skipped 32 previous similar messages [ 319.542199] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 20:07:14 (1746144434) [ 320.176497] Lustre: DEBUG MARKER: SKIP: sanity test_60h Need at least 2 MDTs [ 320.855017] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 321.562250] Lustre: DEBUG MARKER: == sanity test 60j: llog_reader reports corruptions ====== 20:07:16 (1746144436) [ 322.484618] Lustre: DEBUG MARKER: SKIP: sanity test_60j ldiskfs only test [ 323.353120] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 20:07:18 (1746144438) [ 326.504193] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 20:07:21 (1746144441) [ 329.097280] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 20:07:24 (1746144444) [ 392.846605] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 20:08:28 (1746144508) [ 399.639959] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 20:08:34 (1746144514) [ 403.012276] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 403.624938] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 20:08:39 (1746144519) [ 406.811971] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 20:08:42 (1746144522) [ 435.030269] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 20:09:10 (1746144550) [ 436.675935] Lustre: *** cfs_fail_loc=725, val=0*** [ 438.621769] Lustre: *** cfs_fail_loc=725, val=0*** [ 442.407819] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 20:09:17 (1746144557) [ 447.672020] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 20:09:22 (1746144562) [ 487.656749] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 20:10:03 (1746144603) [ 500.328106] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 20:10:15 (1746144615) [ 503.508736] Lustre: *** cfs_fail_loc=513, val=17*** [ 503.510678] LustreError: 17230:0:(service.c:2266:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1830964849496448 [ 504.255616] Lustre: Failing over lustre-OST0000 [ 504.401423] LustreError: 2137:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 504.408168] Lustre: server umount lustre-OST0000 complete [ 505.511865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.520093] LustreError: 17367:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.530025] LustreError: 17367:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 514.370025] LustreError: 17367:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.532930] LustreError: 8454:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.398517] Lustre: 2711:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 517.468939] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 517.481067] mount.lustre (2711) used greatest stack depth: 9712 bytes left [ 518.872713] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 519.945838] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [ 520.106731] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [ 520.107039] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 523.855844] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 524.683836] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 530.075515] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 20:10:45 (1746144645) [ 533.310130] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 20:10:48 (1746144648) [ 536.852293] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 20:10:52 (1746144652) [ 540.361581] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 20:10:55 (1746144655) [ 543.716007] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 20:10:58 (1746144658) [ 547.127372] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 20:11:02 (1746144662) [ 550.216696] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 20:11:05 (1746144665) [ 553.880048] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 20:11:09 (1746144669) [ 557.130412] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 20:11:12 (1746144672) [ 560.959920] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 20:11:16 (1746144676) [ 564.874687] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 20:11:20 (1746144680) [ 565.792307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 565.802579] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 565.808405] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [ 575.396776] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 583.852934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.857620] Lustre: Skipped 1 previous similar message [ 583.860601] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 583.863785] Lustre: Skipped 1 previous similar message [ 583.868837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 583.882886] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [ 583.890498] Lustre: Skipped 1 previous similar message [ 585.101671] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 585.174280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 593.917443] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 602.206308] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.211828] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 602.222360] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 602.229025] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [ 604.103936] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 604.252688] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 607.336160] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 20:12:02 (1746144722) [ 610.569758] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 20:12:05 (1746144725) [ 613.562080] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 20:12:08 (1746144728) [ 620.028164] Lustre: DEBUG MARKER: SKIP: sanity test_65n needs >= 2 MDTs [ 628.255168] Lustre: DEBUG MARKER: == sanity test 65o: pool inheritance for mdt component === 20:12:23 (1746144743) [ 643.419067] Lustre: DEBUG MARKER: == sanity test 65p: setstripe with yaml file and huge number ========================================================== 20:12:38 (1746144758) [ 647.295684] Lustre: DEBUG MARKER: == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 20:12:42 (1746144762) [ 650.812067] Lustre: DEBUG MARKER: == sanity test 65r: prevent all-zero offsets ============= 20:12:46 (1746144766) [ 654.291856] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 20:12:49 (1746144769) [ 659.538797] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 20:12:54 (1746144774) [ 660.273484] Lustre: *** cfs_fail_loc=217, val=0*** [ 660.275692] Lustre: Skipped 4 previous similar messages [ 661.437568] Lustre: *** cfs_fail_loc=217, val=0*** [ 661.439793] Lustre: Skipped 4 previous similar messages [ 664.288791] Lustre: DEBUG MARKER: == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 20:12:59 (1746144779) [ 669.935474] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 670.691891] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 20:13:06 (1746144786) [ 673.654596] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 20:13:09 (1746144789) [ 676.980239] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 20:13:12 (1746144792) [ 705.868927] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 20:13:41 (1746144821) [ 708.447591] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 20:13:43 (1746144823) [ 711.085988] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 20:13:46 (1746144826) [ 713.787755] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 20:13:49 (1746144829) [ 758.441376] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 20:14:33 (1746144873) [ 788.952829] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 20:15:04 (1746144904) [ 792.644847] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 20:15:07 (1746144907) [ 792.803874] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.46@tcp inode [0x200000406:0xc33:0x0] object 0x240000400:3785 extent [0-1048575]: client csum a1034fff, server csum a1034ffe [ 794.064930] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 795.839756] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.46@tcp inode [0x200000406:0xc33:0x0] object 0x240000400:3785 extent [0-1048575], client returned csum 9bb16acb (type 1), server csum ffc34d15 (type 1) [ 796.941685] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 798.115895] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.46@tcp inode [0x200000406:0xc33:0x0] object 0x240000400:3785 extent [0-1048575], client returned csum 762327f5 (type 2), server csum 72cc27f1 (type 2) [ 799.392631] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 800.841517] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.46@tcp inode [0x200000406:0xc33:0x0] object 0x240000400:3785 extent [0-1048575], client returned csum 6a3c5d51 (type 4), server csum a1034ffe (type 4) [ 802.138249] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 803.416156] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.46@tcp inode [0x200000406:0xc33:0x0] object 0x240000400:3785 extent [0-1048575], client returned csum 20a8fb4b (type 10), server csum 5146fc4e (type 10) [ 804.482586] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 806.754622] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 807.946427] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.46@tcp inode [0x200000406:0xc33:0x0] object 0x240000400:3785 extent [0-1048575], client returned csum cd5ee139 (type 40), server csum d66e1ad (type 40) [ 807.971279] LustreError: Skipped 1 previous similar message [ 809.026503] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 811.138847] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 813.878754] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 20:15:29 (1746144929) [ 816.970276] Lustre: 29394:0:(tgt_handler.c:1935:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000406:0xc34:0x0]:[0-1048575]-6a3c5d51-a1034ffe [ 816.984858] LustreError: dumping log to /tmp/lustre-log.1746144933.29394 [ 817.058911] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.46@tcp inode [0x200000406:0xc34:0x0] object 0x240000400:3786 extent [0-1048575], client returned csum 6a3c5d51 (type 4), server csum a1034ffe (type 4) [ 817.075382] LustreError: Skipped 1 previous similar message [ 826.679164] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 20:15:42 (1746144942) [ 826.853846] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.46@tcp inode [0x200000406:0xc36:0x0] object 0x280000400:3777 extent [0-1048575]: client csum b5ea7f3c, server csum b5ea7f3b [ 832.576246] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 20:15:47 (1746144947) [ 833.332891] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 833.421707] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.46@tcp inode [0x200000406:0xc37:0x0] object 0x240000400:3787 extent [0-1048575]: client csum b2f1b12, server csum b2f1b11 [ 833.430346] LustreError: Skipped 1 previous similar message [ 836.703873] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.46@tcp inode [0x200000406:0xc37:0x0] object 0x240000400:3787 extent [4194304-5242879]: client csum b2f1b12, server csum b2f1b11 [ 836.728048] LustreError: Skipped 16 previous similar messages [ 843.136761] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.46@tcp inode [0x200000406:0xc37:0x0] object 0x240000400:3787 extent [4194304-5242879]: client csum b2f1b12, server csum b2f1b11 [ 843.144580] LustreError: Skipped 13 previous similar messages [ 854.112903] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.46@tcp inode [0x200000406:0xc37:0x0] object 0x240000400:3787 extent [4194304-5242879]: client csum b2f1b12, server csum b2f1b11 [ 854.122390] LustreError: Skipped 15 previous similar messages [ 871.041534] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.46@tcp inode [0x200000406:0xc37:0x0] object 0x240000400:3787 extent [2097152-3145727]: client csum b2f1b12, server csum b2f1b11 [ 871.046568] LustreError: Skipped 16 previous similar messages [ 892.663302] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 904.031922] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.46@tcp inode [0x200000406:0xc37:0x0] object 0x240000400:3787 extent [0-1048575]: client csum 19eeae62, server csum 19eeae61 [ 904.062590] LustreError: Skipped 56 previous similar messages [ 950.027145] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 971.134588] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.46@tcp inode [0x200000406:0xc37:0x0] object 0x240000400:3787 extent [6291456-7340031]: client csum b5ea7f3c, server csum b5ea7f3b [ 971.154237] LustreError: Skipped 101 previous similar messages [ 1006.208951] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1062.232939] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1107.364144] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.46@tcp inode [0x200000406:0xc37:0x0] object 0x240000400:3787 extent [0-1048575]: client csum 30ec5402, server csum 30ec5401 [ 1107.370128] LustreError: Skipped 197 previous similar messages [ 1118.447359] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1174.307810] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1230.367856] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1234.162892] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 20:22:29 (1746145349) [ 1235.077373] Lustre: *** cfs_fail_loc=21a, val=0*** [ 1238.211332] Lustre: *** cfs_fail_loc=21b, val=0*** [ 1244.015702] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 20:22:39 (1746145359) [ 1244.481712] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1245.804218] Lustre: Modifying parameter lustre.osc.lustre*.checksums in log params [ 1249.289689] Lustre: Disabling parameter lustre.osc.lustre*.checksums in log params [ 1254.293035] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1256.155225] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 20:22:51 (1746145371) [ 1256.992686] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 1257.847452] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1259.583159] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid 50 [ 1262.534667] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid in IDLE state after 2 sec [ 1264.691988] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid 50 [ 1265.928138] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid in FULL state after 0 sec [ 1267.033576] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1269.201120] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid 50 [ 1282.706973] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid in IDLE state after 12 sec [ 1284.413234] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid 50 [ 1285.231635] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid in FULL state after 0 sec [ 1286.076658] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1287.883211] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid 50 [ 1298.087222] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid in IDLE state after 9 sec [ 1299.909073] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid 50 [ 1300.650835] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid in FULL state after 0 sec [ 1301.316146] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1302.717005] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid 50 [ 1312.914430] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid in IDLE state after 9 sec [ 1314.767329] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid 50 [ 1315.654987] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid in FULL state after 0 sec [ 1316.609414] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1318.545967] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid 50 [ 1327.637967] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid in IDLE state after 8 sec [ 1329.365511] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid 50 [ 1330.196363] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid in FULL state after 0 sec [ 1331.022867] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1332.990647] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid 50 [ 1343.205053] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid in IDLE state after 9 sec [ 1344.838819] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid 50 [ 1345.427841] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid in FULL state after 0 sec [ 1346.235738] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1347.918756] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid 50 [ 1357.998661] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid in IDLE state after 9 sec [ 1359.758224] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid 50 [ 1360.466993] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880136b39800.ost_server_uuid in FULL state after 0 sec [ 1364.512534] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1365.613772] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 20:24:40 (1746145480) [ 1369.409246] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 20:24:44 (1746145484) [ 1370.742292] Lustre: DEBUG MARKER: SKIP: sanity test_77n f77n.sanity blocks not contiguous around hole [ 1371.749091] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 20:24:46 (1746145486) [ 1379.006109] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 20:24:54 (1746145494) [ 1385.790706] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check =========== 20:25:00 (1746145500) [ 1402.572956] Lustre: DEBUG MARKER: sanity test_79: @@@@@@ FAIL: df namelen(255) != llite namelen(256) [ 1410.428395] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 20:25:25 (1746145525) [ 1416.940201] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 20:25:31 (1746145531) [ 1417.973474] Lustre: *** cfs_fail_loc=228, val=0*** [ 1422.515730] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 20:25:37 (1746145537) [ 1423.371288] Lustre: *** cfs_fail_loc=228, val=0*** [ 1427.889010] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 20:25:42 (1746145542) [ 1446.956664] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ========================================================== 20:26:01 (1746145561) [ 1452.073289] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 20:26:06 (1746145566) [ 1551.891402] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 20:27:46 (1746145666) [ 1569.050840] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 20:28:04 (1746145684) [ 1610.353556] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 20:28:45 (1746145725) [ 1833.244771] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 20:32:28 (1746145948) [ 1874.586974] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 20:33:09 (1746145989) [ 1880.088415] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 20:33:15 (1746145995) [ 1949.916448] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 20:34:25 (1746146065) [ 1959.535899] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 20:34:34 (1746146074) [ 1963.933266] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 20:34:39 (1746146079) [ 1997.190667] Lustre: DEBUG MARKER: == sanity test 101m: read ahead for small file and last stripe of the file ========================================================== 20:35:12 (1746146112) [ 1998.043691] Lustre: DEBUG MARKER: SKIP: sanity test_101m need >= 2.13.57 and ldiskfs for fallocate [ 1998.939032] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 20:35:14 (1746146114) [ 2002.452689] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 20:35:17 (1746146117) [ 2008.969038] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 20:35:24 (1746146124) [ 2012.148281] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 20:35:27 (1746146127) [ 2030.762856] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 20:35:46 (1746146146) [ 2053.818364] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 20:36:08 (1746146168) [ 2054.731184] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 2055.645219] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 2056.490931] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 2057.317125] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2060.790208] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 20:36:15 (1746146175) [ 2062.085790] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2063.173599] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 2064.213758] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 2065.272781] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2066.460628] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2070.241508] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 20:36:25 (1746146185) [ 2074.634432] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 20:36:29 (1746146189) [ 2096.164899] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 20:36:51 (1746146211) [ 2099.300752] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 20:36:54 (1746146214) [ 2102.269637] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 20:36:57 (1746146217) [ 2105.021952] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 20:37:00 (1746146220) [ 2108.954192] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 20:37:04 (1746146224) [ 2112.251106] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 20:37:07 (1746146227) [ 2115.206145] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 20:37:10 (1746146230) [ 2119.273737] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 20:37:14 (1746146234) [ 2122.491274] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 20:37:17 (1746146237) [ 2126.120245] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 20:37:21 (1746146241) [ 2239.777176] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 20:39:15 (1746146355) [ 2314.880788] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 20:40:30 (1746146430) [ 2317.575723] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 20:40:32 (1746146432) [ 2703.165300] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 20:46:58 (1746146818) [ 2704.193117] Lustre: lustre-MDD0000: changelog on [ 2707.075966] Lustre: lustre-MDD0000: changelog off [ 2708.061803] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 20:47:03 (1746146823) [ 2708.246619] Lustre: lustre-OST0000: Client 3d355148-a95c-47c3-a91e-da35553b07a8 (at 192.168.203.46@tcp) reconnecting [ 2709.369048] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012d176800.ost_server_uuid 50 [ 2709.863563] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012d176800.ost_server_uuid in FULL state after 0 sec [ 2713.034369] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 20:47:08 (1746146828) [ 2715.776013] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 20:47:11 (1746146831) [ 2726.301107] Lustre: DEBUG MARKER: == sanity test 104d: runas -u 500 -g 500 lctl dl test ==== 20:47:21 (1746146841) [ 2728.569498] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 20:47:24 (1746146844) [ 2730.796165] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 20:47:26 (1746146846) [ 2733.162559] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 20:47:28 (1746146848) [ 2735.342647] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 20:47:30 (1746146850) [ 2747.603008] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 20:47:43 (1746146863) [ 2750.012752] Lustre: DEBUG MARKER: == sanity test 105f: Enqueue same range flocks =========== 20:47:45 (1746146865) [ 2753.304123] Lustre: DEBUG MARKER: == sanity test 105g: ldlm_lock_debug stack test ========== 20:47:48 (1746146868) [ 2757.962934] Lustre: DEBUG MARKER: == sanity test 105h: Flock functional verify ============= 20:47:53 (1746146873) [ 2760.617788] Lustre: DEBUG MARKER: == sanity test 105i: Flock deadlock verify =============== 20:47:56 (1746146876) [ 2769.076617] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 20:48:04 (1746146884) [ 2771.993911] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 20:48:07 (1746146887) [ 2776.609826] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 20:48:11 (1746146891) [ 2779.577507] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 20:48:14 (1746146894) [ 2929.891444] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 20:50:45 (1746147045) [ 2931.460911] Lustre: *** cfs_fail_loc=147, val=0*** [ 2935.841774] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 20:50:51 (1746147051) [ 2939.198714] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 20:50:54 (1746147054) [ 2942.866492] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 20:50:58 (1746147058) [ 2943.934120] Lustre: *** cfs_fail_loc=217, val=0*** [ 2947.987527] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 2948.701748] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 2949.430597] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 20:51:04 (1746147064) [ 2952.902055] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 20:51:08 (1746147068) [ 2956.269699] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 20:51:11 (1746147071) [ 2957.282415] Lustre: *** cfs_fail_loc=20e, val=0*** [ 2958.217133] Lustre: *** cfs_fail_loc=20e, val=0*** [ 2960.230651] Lustre: *** cfs_fail_loc=20e, val=0*** [ 2963.195463] Lustre: *** cfs_fail_loc=20e, val=0*** [ 2967.214249] Lustre: *** cfs_fail_loc=20e, val=0*** [ 2971.375457] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 20:51:26 (1746147086) [ 2975.224951] Lustre: *** cfs_fail_loc=20e, val=0*** [ 2975.230037] Lustre: Skipped 2 previous similar messages [ 2981.208797] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 20:51:36 (1746147096) [ 2985.067753] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 20:51:40 (1746147100) [ 2996.095381] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 20:51:51 (1746147111) [ 2999.518634] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 20:51:54 (1746147114) [ 3003.081012] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 20:51:58 (1746147118) [ 3009.039853] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 20:52:04 (1746147124) [ 3011.870649] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 20:52:07 (1746147127) [ 3014.751516] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 20:52:10 (1746147130) [ 3017.466948] Lustre: DEBUG MARKER: == sanity test 119e: Basic tests of dio read and write at various sizes ========================================================== 20:52:12 (1746147132) [ 3019.295933] Lustre: DEBUG MARKER: sanity test_119e: @@@@@@ FAIL: trivial unaligned dio failed [ 3025.361845] Lustre: DEBUG MARKER: == sanity test 119f: dio vs dio race ===================== 20:52:20 (1746147140) [ 3038.663189] Lustre: DEBUG MARKER: sanity test_119f: @@@@@@ FAIL: dio copy 1 w/bsize 1049600 failed: 1 [ 3046.618258] Lustre: DEBUG MARKER: == sanity test 119g: dio vs buffered I/O race ============ 20:52:41 (1746147161) [ 3062.080837] Lustre: DEBUG MARKER: sanity test_119g: @@@@@@ FAIL: dio copy 1 w/bsize 1049600 failed: 1 [ 3069.191565] Lustre: DEBUG MARKER: == sanity test 119h: basic tests of memory unaligned dio ========================================================== 20:53:04 (1746147184) [ 3070.853966] Lustre: DEBUG MARKER: sanity test_119h: @@@@@@ FAIL: multiop memory unaligned write failed, 1044480 [ 3077.741652] Lustre: DEBUG MARKER: SKIP: sanity test_119i skipping ALWAYS excluded test 119i [ 3078.677879] Lustre: DEBUG MARKER: == sanity test 119j: basic tests of hybrid IO switching == 20:53:13 (1746147193) [ 3079.420640] Lustre: DEBUG MARKER: SKIP: sanity test_119j needs kernel > 4.5.0 for ki_flags support [ 3080.330104] Lustre: DEBUG MARKER: == sanity test 119m: Test DIO readv/writev: exercise iter duplication ========================================================== 20:53:15 (1746147195) [ 3084.259510] Lustre: DEBUG MARKER: == sanity test 119n: Test Unaligned DIO readv() and writev() with unpatched ZFS ========================================================== 20:53:19 (1746147199) [ 3085.158848] Lustre: DEBUG MARKER: SKIP: sanity test_119n zfs server without 'unaligned_dio' support [ 3086.062287] Lustre: DEBUG MARKER: == sanity test 119o: Test Unaligned DIO readv() and writev() with unpatched servers ========================================================== 20:53:21 (1746147201) [ 3086.929414] Lustre: DEBUG MARKER: SKIP: sanity test_119o need ldiskfs without 'unaligned_dio' support [ 3087.789220] Lustre: DEBUG MARKER: == sanity test 119p: Test Unaligned DIO readv() and writev() with patched servers ========================================================== 20:53:23 (1746147203) [ 3088.554086] Lustre: DEBUG MARKER: sanity test_119p: @@@@@@ FAIL: DIO unaligned writev test failed [ 3094.702466] Lustre: DEBUG MARKER: == sanity test 119q: Test patchded Unaligned DIO readv() and writev() ========================================================== 20:53:29 (1746147209) [ 3095.595971] Lustre: DEBUG MARKER: sanity test_119q: @@@@@@ FAIL: DIO unaligned writev test failed: 69631 [ 3101.154448] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 20:53:36 (1746147216) [ 3106.869710] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 20:53:41 (1746147221) [ 3112.345615] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 20:53:47 (1746147227) [ 3117.709604] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 20:53:52 (1746147232) [ 3122.916957] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 20:53:58 (1746147238) [ 3135.281773] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 20:54:10 (1746147250) [ 3146.844050] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 20:54:22 (1746147262) [ 3378.820793] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 20:58:14 (1746147494) [ 3381.331380] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 20:58:16 (1746147496) [ 3385.397874] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 3386.634441] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 0 sec [ 3404.653106] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 11 sec [ 3407.393259] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 1 sec [ 3470.660214] Lustre: DEBUG MARKER: ls -l 4000 files without statahead: 45 sec [ 3481.792674] Lustre: DEBUG MARKER: ls -l 4000 files with statahead: 9 sec [ 3482.405375] Lustre: DEBUG MARKER: ls -l done [ 3521.570961] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity: 38 seconds [ 3524.764751] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 21:00:40 (1746147640) [ 3525.395158] Lustre: DEBUG MARKER: SKIP: sanity test_123ab Test must be statx() syscall supported [ 3526.132029] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 21:00:41 (1746147641) [ 3526.924397] Lustre: DEBUG MARKER: SKIP: sanity test_123ac Test must be statx() syscall supported [ 3527.716879] Lustre: DEBUG MARKER: == sanity test 123ad: Verify batching statahead works correctly ========================================================== 21:00:43 (1746147643) [ 3532.063837] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 3533.049682] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 0 sec [ 3549.775744] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 10 sec [ 3553.186485] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 2 sec [ 3633.305655] Lustre: DEBUG MARKER: ls -l 5000 files without statahead: 59 sec [ 3652.250825] Lustre: DEBUG MARKER: ls -l 5000 files with statahead: 16 sec [ 3652.903263] Lustre: DEBUG MARKER: ls -l done [ 3699.996641] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 46 seconds [ 4013.833375] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 2 sec [ 4015.362684] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 4038.365573] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 12 sec [ 4041.451414] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 2 sec [ 4120.350609] Lustre: DEBUG MARKER: ls -l 4000 files without statahead: 54 sec [ 4131.123231] Lustre: DEBUG MARKER: ls -l 4000 files with statahead: 8 sec [ 4132.006000] Lustre: DEBUG MARKER: ls -l done [ 4178.747521] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 46 seconds [ 4431.067946] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 21:15:46 (1746148546) [ 4447.495757] Lustre: DEBUG MARKER: ls done [ 4462.335407] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 21:16:17 (1746148577) [ 4463.227085] Lustre: DEBUG MARKER: SKIP: sanity test_123c needs >= 2 MDTs [ 4464.265060] Lustre: DEBUG MARKER: == sanity test 123d: Statahead on striped directories works correctly ========================================================== 21:16:19 (1746148579) [ 4471.641509] Lustre: DEBUG MARKER: == sanity test 123e: statahead with large wide striping == 21:16:26 (1746148586) [ 4545.384109] Lustre: 26520:0:(mdt_handler.c:4672:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 4605.150933] Lustre: DEBUG MARKER: == sanity test 123f: Retry mechanism with large wide striping files ========================================================== 21:18:39 (1746148719) [ 4659.775856] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x280000400 to 0x280000401 [ 4659.941347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x240000400 to 0x240000401 [ 4764.373756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x240000401 to 0x240000402 [ 4774.915894] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4790.192028] Lustre: lustre-MDT0000: Client 88e4e19c-2e65-418a-97a9-bbcebf75e565 (at 192.168.203.46@tcp) reconnecting [ 5072.438177] Lustre: DEBUG MARKER: == sanity test 123g: Test for stat-ahead advise ========== 21:26:27 (1746149187) [ 5112.438156] Lustre: DEBUG MARKER: == sanity test 123h: Verify statahead work with the fname pattern via du ========================================================== 21:27:07 (1746149227) [ 6118.457165] Lustre: DEBUG MARKER: == sanity test 123i: Verify statahead work with the fname indexing pattern ========================================================== 21:43:53 (1746150233) [ 6218.332402] Lustre: DEBUG MARKER: == sanity test 123j: -ENOENT error from batched statahead be handled correctly ========================================================== 21:45:33 (1746150333) [ 6219.241448] Lustre: DEBUG MARKER: SKIP: sanity test_123j needs >= 2 MDTs [ 6220.634885] Lustre: DEBUG MARKER: == sanity test 123k: Verify statahead work with mdtest shared stat() mode ========================================================== 21:45:35 (1746150335) [ 6221.771517] Lustre: DEBUG MARKER: SKIP: sanity test_123k mdtest not found [ 6223.045714] Lustre: DEBUG MARKER: == sanity test 123l: Avoid panic when revalidate a local cached entry ========================================================== 21:45:38 (1746150338) [ 6298.472589] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 21:46:53 (1746150413) [ 6299.292857] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 6327.672430] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ab3bb800 [ 6328.628756] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ab3bb800 [ 6329.617203] Lustre: DEBUG MARKER: LRU=2003 [ 6330.789292] Lustre: DEBUG MARKER: LIMIT=62166 [ 6331.693457] Lustre: DEBUG MARKER: LVF=3724300 [ 6332.790112] Lustre: DEBUG MARKER: OLD_LVF=100 [ 6333.731013] Lustre: DEBUG MARKER: Sleep 50 sec [ 6384.843179] Lustre: DEBUG MARKER: Dropped 870 locks in 50s [ 6385.952099] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 6405.151646] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 21:48:40 (1746150520) [ 6476.153945] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 6606.876747] Lustre: DEBUG MARKER: ls -la time: 129 seconds [ 6607.751457] Lustre: DEBUG MARKER: lru_size = 400 [ 6758.186839] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 6810.487384] Lustre: DEBUG MARKER: ls -la time: 49 seconds [ 6811.454583] Lustre: DEBUG MARKER: lru_size = 8005 [ 6812.473002] Lustre: DEBUG MARKER: ls -la is 62% faster with lru resize enabled [ 6864.127538] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 21:56:19 (1746150979) [ 6892.283589] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize disabled ========================================================== 21:56:47 (1746151007) [ 6920.638813] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 21:57:15 (1746151035) [ 6924.386450] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 21:57:19 (1746151039) [ 6927.769208] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 21:57:23 (1746151043) [ 6931.323107] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 21:57:26 (1746151046) [ 6934.967894] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [ 7052.565720] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 21:59:27 (1746151167) [ 7055.781934] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 21:59:31 (1746151171) [ 7056.752537] Lustre: DEBUG MARKER: SKIP: sanity test_129 ldiskfs only test [ 7057.595492] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 21:59:32 (1746151172) [ 7058.343940] Lustre: DEBUG MARKER: SKIP: sanity test_130a LU-1941: FIEMAP unimplemented on ZFS [ 7059.237206] Lustre: DEBUG MARKER: SKIP: sanity test_130b skipping ALWAYS excluded test 130b [ 7059.940658] Lustre: DEBUG MARKER: SKIP: sanity test_130c skipping ALWAYS excluded test 130c [ 7060.719220] Lustre: DEBUG MARKER: SKIP: sanity test_130d skipping ALWAYS excluded test 130d [ 7061.520964] Lustre: DEBUG MARKER: SKIP: sanity test_130e skipping ALWAYS excluded test 130e [ 7062.277557] Lustre: DEBUG MARKER: SKIP: sanity test_130f skipping ALWAYS excluded test 130f [ 7062.993686] Lustre: DEBUG MARKER: SKIP: sanity test_130g skipping ALWAYS excluded test 130g [ 7063.730814] Lustre: DEBUG MARKER: == sanity test 130h: FIEMAP deadlock ===================== 21:59:39 (1746151179) [ 7072.792096] Lustre: DEBUG MARKER: == sanity test 130i: FIEMAP (DoM file) =================== 21:59:48 (1746151188) [ 7073.533779] Lustre: DEBUG MARKER: SKIP: sanity test_130i LU-1941: FIEMAP unimplemented on ZFS [ 7074.300909] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 21:59:49 (1746151189) [ 7077.556433] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 21:59:52 (1746151192) [ 7081.220413] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 21:59:56 (1746151196) [ 7084.923157] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 22:00:00 (1746151200) [ 7089.258568] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 22:00:04 (1746151204) [ 7093.434589] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 22:00:08 (1746151208) [ 7105.574183] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 22:00:20 (1746151220) [ 7119.817104] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 22:00:34 (1746151234) [ 7149.769120] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 22:01:05 (1746151265) [ 7171.350778] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 22:01:26 (1746151286) [ 7179.302943] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 22:01:34 (1746151294) [ 7191.096170] LustreError: 24044:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 7191.099304] LustreError: 24044:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7191.262982] Lustre: server umount lustre-MDT0000 complete [ 7193.250680] LustreError: 15983:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746151309 with bad export cookie 16733967779768520261 [ 7193.256628] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7203.345109] LustreError: 24423:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 7203.348110] LustreError: 24423:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7203.376769] Lustre: server umount lustre-OST0000 complete [ 7215.442575] LustreError: 24746:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7215.449234] LustreError: 24746:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7215.550720] Lustre: server umount lustre-OST0001 complete [ 7220.425854] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 7221.487836] Key type lgssc unregistered [ 7221.615897] LNet: 25307:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7222.630587] LNet: Removed LNI 192.168.203.146@tcp [ 7228.383257] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 7228.552614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7228.564479] alg: No test for adler32 (adler32-zlib) [ 7229.422008] Lustre: Lustre: Build Version: 2.16.54_72_gbbb30b7 [ 7229.491192] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 7229.493994] LNet: Accept secure, port 988 [ 7231.053444] Key type lgssc registered [ 7231.394435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7236.251937] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 7239.494985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7240.914054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7242.411755] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [ 7243.964430] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 7246.755844] Lustre: 28124:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7246.997245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7249.706033] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [ 7250.982482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000402:37109 to 0x240000402:37249) [ 7253.516831] Lustre: 29179:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7253.658105] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7256.275104] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [ 7258.168974] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:30673 to 0x280000402:30689) [ 7261.507930] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7263.171571] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 7272.285048] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 22:03:07 (1746151387) [ 7287.097012] LNet: 31641:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 7287.119635] Lustre: DEBUG MARKER:  [ 7287.120906] Lustre: DEBUG MARKER:  [ 7300.132006] LNet: 308:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 7300.136135] LNet: 308:0:(debug.c:388:cfs_str2mask()) Skipped 3 previous similar messages [ 7300.158278] Lustre: DEBUG MARKER:  [ 7300.159538] Lustre: DEBUG MARKER:  [ 7303.720812] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7303.726834] Lustre: Skipped 1 previous similar message [ 7303.728259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7304.206227] LustreError: 554:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7304.278150] Lustre: server umount lustre-MDT0000 complete [ 7306.129983] LustreError: 27118:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746151422 with bad export cookie 344616308372989066 [ 7306.131480] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7306.147286] LustreError: 27118:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7308.160487] LustreError: 967:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7308.165072] LustreError: 967:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7308.181510] Lustre: server umount lustre-OST0000 complete [ 7310.072684] LustreError: 1289:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7310.075518] LustreError: 1289:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7310.161976] Lustre: server umount lustre-OST0001 complete [ 7316.348243] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 7317.335260] Key type lgssc unregistered [ 7317.453280] LNet: 1913:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7317.473143] LNet: Removed LNI 192.168.203.146@tcp [ 7323.850019] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 7324.026179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7324.072574] alg: No test for adler32 (adler32-zlib) [ 7324.911044] Lustre: Lustre: Build Version: 2.16.54_72_gbbb30b7 [ 7324.979213] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 7324.983362] LNet: Accept secure, port 988 [ 7326.532980] Key type lgssc registered [ 7326.964061] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7332.039177] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 7335.409646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7336.867882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7338.591031] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [ 7340.184208] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 7342.868560] Lustre: 4754:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7343.118226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7343.210605] mount.lustre (4754) used greatest stack depth: 9392 bytes left [ 7345.662252] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [ 7346.109926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000402:37109 to 0x240000402:37281) [ 7348.847630] Lustre: 5792:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7348.997010] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7351.410103] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [ 7354.003345] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:30673 to 0x280000402:30721) [ 7356.234814] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7357.758892] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 7366.725456] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 22:04:41 (1746151481) [ 7762.573443] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 22:11:17 (1746151877) [ 7772.817955] Lustre: *** cfs_fail_loc=327, val=500*** [ 7795.176309] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 22:11:50 (1746151910) [ 7798.244728] Lustre: *** cfs_fail_loc=328, val=500*** [ 7798.248169] Lustre: Skipped 513 previous similar messages [ 7799.245037] Lustre: *** cfs_fail_loc=328, val=500*** [ 7799.246148] Lustre: Skipped 150 previous similar messages [ 7801.246944] Lustre: *** cfs_fail_loc=328, val=500*** [ 7801.248427] Lustre: Skipped 297 previous similar messages [ 7801.594539] Lustre: 19270:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8800616bea00 x1830972488296064/t0(0) o101->8986ab61-9359-4719-b3a8-4337866f1550@192.168.203.46@tcp:8/0 lens 648/0 e 0 to 0 dl 1746151928 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7802.623267] Lustre: 19270:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8800616bf800 x1830972488296832/t0(0) o101->8986ab61-9359-4719-b3a8-4337866f1550@192.168.203.46@tcp:9/0 lens 648/0 e 0 to 0 dl 1746151929 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7803.658023] Lustre: 19270:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8800616bed80 x1830972488297600/t0(0) o101->8986ab61-9359-4719-b3a8-4337866f1550@192.168.203.46@tcp:10/0 lens 648/0 e 0 to 0 dl 1746151930 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7805.711158] Lustre: *** cfs_fail_loc=328, val=500*** [ 7805.713843] Lustre: Skipped 55 previous similar messages [ 7805.717233] Lustre: 3745:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff88008c239880 x1830972488299520/t0(0) o101->8986ab61-9359-4719-b3a8-4337866f1550@192.168.203.46@tcp:12/0 lens 648/0 e 0 to 0 dl 1746151932 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7805.731269] Lustre: 3745:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) Skipped 1 previous similar message [ 7809.801112] Lustre: 3744:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8800a6ea3800 x1830972488302976/t0(0) o101->8986ab61-9359-4719-b3a8-4337866f1550@192.168.203.46@tcp:16/0 lens 648/0 e 0 to 0 dl 1746151936 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7809.813130] Lustre: 3744:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) Skipped 3 previous similar messages [ 7813.888760] Lustre: *** cfs_fail_loc=328, val=500*** [ 7813.890451] Lustre: Skipped 7 previous similar messages [ 7818.777941] LustreError: 4901:0:(ldlm_lockd.c:3258:lock_reclaim_threshold_mb_store()) Failed to set lock_reclaim_threshold_mb, rc = -22. [ 7827.954933] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [ 7828.939754] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [ 7829.705792] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 22:12:24 (1746151944) [ 7848.027053] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 22:12:43 (1746151963) [ 7885.289168] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 22:13:20 (1746152000) [ 7886.281642] Lustre: DEBUG MARKER: SKIP: sanity test_150b need >= 2.13.57 and ldiskfs for fallocate [ 7887.431210] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 22:13:22 (1746152002) [ 7888.283308] Lustre: DEBUG MARKER: SKIP: sanity test_150bb need >= 2.13.57 and ldiskfs for fallocate [ 7889.218855] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 22:13:24 (1746152004) [ 7889.980529] Lustre: DEBUG MARKER: SKIP: sanity test_150c need >= 2.13.57 and ldiskfs for fallocate [ 7891.052113] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 22:13:26 (1746152006) [ 7891.939213] Lustre: DEBUG MARKER: SKIP: sanity test_150d need >= 2.13.57 and ldiskfs for fallocate [ 7893.150822] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 22:13:28 (1746152008) [ 7894.173553] Lustre: DEBUG MARKER: SKIP: sanity test_150e need >= 2.13.57 and ldiskfs for fallocate [ 7895.251895] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 22:13:30 (1746152010) [ 7896.060820] Lustre: DEBUG MARKER: SKIP: sanity test_150f LU-14160: punch mode is not implemented on OSD ZFS [ 7897.087041] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 22:13:32 (1746152012) [ 7897.968414] Lustre: DEBUG MARKER: SKIP: sanity test_150g LU-14160: punch mode is not implemented on OSD ZFS [ 7899.081124] Lustre: DEBUG MARKER: == sanity test 150h: Verify extend fallocate updates the file size ========================================================== 22:13:34 (1746152014) [ 7899.928269] Lustre: DEBUG MARKER: SKIP: sanity test_150h need >= 2.13.57 and ldiskfs for fallocate [ 7900.993525] Lustre: DEBUG MARKER: == sanity test 150ia: Verify fallocate zero-range ZERO functionality ========================================================== 22:13:36 (1746152016) [ 7901.913532] Lustre: DEBUG MARKER: SKIP: sanity test_150ia zero-range mode is not implemented on OSD ZFS [ 7902.937782] Lustre: DEBUG MARKER: == sanity test 150ib: Verify fallocate zero-range PREALLOC functionality ========================================================== 22:13:38 (1746152018) [ 7903.747131] Lustre: DEBUG MARKER: SKIP: sanity test_150ib zero-range mode is not implemented on OSD ZFS [ 7904.770015] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 22:13:39 (1746152019) [ 7906.376496] Lustre: DEBUG MARKER: SKIP: sanity test_151 not cache-capable obdfilter [ 7907.390598] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 22:13:42 (1746152022) [ 7911.146383] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 22:13:46 (1746152026) [ 7914.561171] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 22:13:49 (1746152029) [ 7917.914295] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 22:13:53 (1746152033) [ 7921.575098] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 22:13:56 (1746152036) [ 7922.475670] LustreError: 3744:0:(fld_handler.c:244:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 7926.993966] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 22:14:02 (1746152042) [ 7927.752605] Lustre: DEBUG MARKER: SKIP: sanity test_154b needs >= 2 MDTs [ 7928.772469] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 22:14:03 (1746152043) [ 7932.382109] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 22:14:07 (1746152047) [ 7936.252616] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 22:14:11 (1746152051) [ 7939.823397] Lustre: DEBUG MARKER: == sanity test 154ea: .lustre is not returned by readdir (2) ========================================================== 22:14:14 (1746152054) [ 7974.122130] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 22:14:49 (1746152089) [ 7978.722572] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 22:14:53 (1746152093) [ 8590.123220] Lustre: DEBUG MARKER: == sanity test 154h: Verify interactive path2fid ========= 22:25:05 (1746152705) [ 8593.515896] Lustre: DEBUG MARKER: == sanity test 154i: fid2path for path longer than PATH_MAX ========================================================== 22:25:08 (1746152708) [ 8614.968031] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 22:25:30 (1746152730) [ 8621.491242] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 22:25:36 (1746152736) [ 8627.888367] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 22:25:42 (1746152742) [ 8635.198277] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 22:25:50 (1746152750) [ 8641.749874] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 22:25:56 (1746152756) [ 8674.379475] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 22:26:29 (1746152789) [ 8717.967636] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 22:27:13 (1746152833) [ 8750.209009] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 22:27:45 (1746152865) [ 8792.466041] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 22:28:27 (1746152907) [ 8793.413402] Lustre: DEBUG MARKER: SKIP: sanity test_156 LU-1956/LU-2261: stats not implemented on OSD ZFS [ 8794.431259] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 22:28:29 (1746152909) [ 8795.978249] Lustre: lustre-MDD0000: changelog on [ 8804.564367] Lustre: Failing over lustre-MDT0000 [ 8804.735842] LustreError: 24664:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8804.872229] Lustre: server umount lustre-MDT0000 complete [ 8808.695053] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8808.911365] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8809.077791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8809.130928] Lustre: lustre-MDD0000: changelog on [ 8809.141882] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8810.900928] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [ 8811.527220] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 8813.985801] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [ 8872.274694] Lustre: lustre-MDT0000: Recovery over after 1:01, of 1 clients 1 recovered and 0 were evicted. [ 8872.314482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000402:38110 to 0x240000402:38145) [ 8872.315961] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:31545 to 0x280000402:31585) [ 8872.699576] Lustre: lustre-MDD0000: changelog off [ 8879.537261] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 22:29:54 (1746152994) [ 8881.226647] Lustre: lustre-MDD0000: changelog on [ 8886.361185] Lustre: lustre-MDD0000: changelog off [ 8888.106936] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 22:30:03 (1746153003) [ 8889.663579] Lustre: lustre-MDD0000: changelog on [ 8894.710861] Lustre: lustre-MDD0000: changelog off [ 8896.161037] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 22:30:11 (1746153011) [ 8897.087366] Lustre: DEBUG MARKER: SKIP: sanity test_160d needs >= 2 MDTs [ 8898.138011] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 22:30:13 (1746153013) [ 8899.730096] Lustre: lustre-MDD0000: changelog on [ 8903.623594] Lustre: lustre-MDD0000: changelog off [ 8904.963916] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 22:30:20 (1746153020) [ 8908.547936] Lustre: DEBUG MARKER: 1746153023: creating first dirs [ 8919.910901] Lustre: *** cfs_fail_loc=1313, val=3*** [ 8919.914868] Lustre: 25345:0:(mdd_dir.c:911:mdd_changelog_emrg_cleanup()) lustre-MDD0000: changelog has only 3 free catalog entries [ 8919.919098] Lustre: 25345:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 8919.927976] Lustre: 30726:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl6 idle for 13s with 4 unprocessed records [ 8927.419452] Lustre: lustre-MDD0000: changelog off [ 8928.478064] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 22:30:43 (1746153043) [ 8929.703083] Lustre: lustre-MDD0000: changelog on [ 8929.706276] Lustre: Skipped 1 previous similar message [ 8935.636766] Lustre: 25824:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 8935.643463] Lustre: 435:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl8 idle for 5s with 4 unprocessed records [ 8942.153109] Lustre: lustre-MDD0000: changelog off [ 8943.162618] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 22:30:58 (1746153058) [ 8960.466965] Lustre: *** cfs_fail_loc=1316, val=0*** [ 8960.468778] Lustre: 1680:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: simulate starting changelog garbage collection [ 8960.479536] Lustre: 2670:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl10 idle for 15s with 4 unprocessed records [ 8961.433909] Lustre: Failing over lustre-MDT0000 [ 8962.770587] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.46@tcp (stopping) [ 8964.051917] LustreError: 2866:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8964.054954] LustreError: 2866:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8964.130077] Lustre: server umount lustre-MDT0000 complete [ 8967.170876] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8967.335209] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8967.341623] Lustre: Skipped 2 previous similar messages [ 8967.410229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8967.454517] Lustre: lustre-MDD0000: changelog on [ 8967.455629] Lustre: Skipped 1 previous similar message [ 8967.457813] Lustre: 3980:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 34 to index 35, being cleared now [ 8967.471801] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8967.760810] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 8968.464160] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 8968.487355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000402:38110 to 0x240000402:38177) [ 8968.492291] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:31587 to 0x280000402:31617) [ 8968.723641] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [ 8972.379854] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.203.146@tcp (at 0@lo) [ 8972.386696] Lustre: Skipped 1 previous similar message [ 8975.956308] Lustre: lustre-MDD0000: changelog off [ 8977.195152] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 22:31:32 (1746153092) [ 8980.304687] LustreError: 6145:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_race id 1315 sleeping [ 8980.383432] Lustre: 2339:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746153080/real 1746153080] req@ffff88009a723100 x1830972494191104/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746153096 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 8982.493274] LustreError: 6314:0:(mdd_device.c:1746:mdd_changelog_user_register()) cfs_fail_race id 1315 waking [ 8982.499290] LustreError: 6145:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_fail_race id 1315 awake: rc=2818 [ 8988.495847] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 22:31:43 (1746153103) [ 8996.090983] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 22:31:51 (1746153111) [ 8997.930560] LustreError: 3993:0:(mdd_dir.c:966:mdd_changelog_store()) cfs_fail_timeout id 15d sleeping for 3000ms [ 9000.936402] LustreError: 3993:0:(mdd_dir.c:966:mdd_changelog_store()) cfs_fail_timeout id 15d awake [ 9008.710440] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 22:32:04 (1746153124) [ 9009.942494] Lustre: lustre-MDD0000: changelog on [ 9009.944500] Lustre: Skipped 4 previous similar messages [ 9015.522948] Lustre: lustre-MDD0000: changelog off [ 9015.524955] Lustre: Skipped 4 previous similar messages [ 9016.358739] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 22:32:11 (1746153131) [ 9020.040866] LustreError: 3993:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_race id 15f sleeping [ 9022.037868] LustreError: 8703:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_fail_race id 15f waking [ 9022.041336] LustreError: 3993:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_fail_race id 15f awake: rc=3004 [ 9026.703307] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 22:32:22 (1746153142) [ 9969.907865] LustreError: 3993:0:(mdd_device.c:408:llog_changelog_cancel_cb()) cfs_race id 16c sleeping [ 9971.908810] LustreError: 8703:0:(llog_osd.c:1136:llog_osd_next_block()) cfs_fail_race id 16c waking [ 9971.913304] LustreError: 3993:0:(mdd_device.c:408:llog_changelog_cancel_cb()) cfs_fail_race id 16c awake: rc=3001 [ 9972.866176] LustreError: 3993:0:(osd_handler.c:210:osd_trans_start()) lustre-MDT0000: can't assign tx: rc = -2 [ 9978.872137] Lustre: lustre-MDD0000: changelog off [ 9978.873972] Lustre: Skipped 1 previous similar message [ 9979.702494] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 22:48:15 (1746154095) [ 9980.624251] Lustre: lustre-MDD0000: changelog on [ 9980.625823] Lustre: Skipped 2 previous similar messages [ 9980.955743] LustreError: 19639:0:(mdd_device.c:1700:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [ 9981.290715] Lustre: 19723:0:(mdd_device.c:1717:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [ 9981.604847] LustreError: 19787:0:(mdd_device.c:1709:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [ 9988.971935] Lustre: lustre-MDD0000: changelog off [ 9990.776040] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 22:48:26 (1746154106) [ 9991.494993] Lustre: DEBUG MARKER: SKIP: sanity test_160p ldiskfs only test [ 9992.122961] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 22:48:27 (1746154107) [ 9993.068570] Lustre: lustre-MDD0000: changelog on [ 9996.059721] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records * time ========================================================== 22:48:31 (1746154111) [10002.312943] Lustre: 3991:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [10002.320412] Lustre: 22743:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl24 idle for 864005s with 500000004 unprocessed records [10008.464156] Lustre: DEBUG MARKER: == sanity test 160t: changelog garbage collect on lack of space ========================================================== 22:48:43 (1746154123) [10009.683790] Lustre: lustre-MDD0000: changelog on [10009.687192] Lustre: Skipped 1 previous similar message [10039.418120] Lustre: *** cfs_fail_loc=18c, val=1212108*** [10039.424652] Lustre: 12709:0:(mdd_dir.c:881:mdd_changelog_is_space_safe()) lustre-MDD0000: changelog uses 33MB with 1MB space limit [10039.430019] Lustre: 12709:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [10039.435760] Lustre: 25343:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl25-user1 idle for 30s with 7506 unprocessed records [10039.916623] Lustre: *** cfs_fail_loc=18c, val=1212108*** [10045.808127] Lustre: lustre-MDD0000: changelog off [10045.809982] Lustre: Skipped 2 previous similar messages [10056.389621] Lustre: DEBUG MARKER: == sanity test 160u: changelog rename record type name and sname strings are correct ========================================================== 22:49:31 (1746154171) [10058.936348] Lustre: lustre-MDD0000: changelog on [10063.963162] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 22:49:39 (1746154179) [10082.280916] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 22:49:57 (1746154197) [10082.881024] Lustre: DEBUG MARKER: SKIP: sanity test_161b skipping remote directory test [10083.526303] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 22:49:58 (1746154198) [10088.312113] Lustre: lustre-MDD0000: changelog off [10088.313678] Lustre: Skipped 1 previous similar message [10089.278241] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 22:50:04 (1746154204) [10096.091796] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 22:50:11 (1746154211) [10099.583377] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 22:50:14 (1746154214) [10100.177967] Lustre: DEBUG MARKER: SKIP: sanity test_162b needs >= 2 MDTs [10100.921068] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 22:50:16 (1746154216) [10123.210789] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 22:50:38 (1746154238) [10129.765026] Lustre: Failing over lustre-OST0000 [10129.846721] LustreError: 31100:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10129.848679] LustreError: 31100:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10129.860440] Lustre: server umount lustre-OST0000 complete [10130.580716] LustreError: 3992:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10130.601237] LustreError: 3992:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [10134.232830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10134.237199] LustreError: 3989:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10135.569011] LustreError: 3989:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10137.156197] Lustre: 32240:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [10137.300166] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10137.310374] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10138.607481] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10139.254273] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10139.254760] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [10139.254761] Lustre: Skipped 1 previous similar message [10139.332757] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [10140.897604] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 22:50:56 (1746154256) [10166.027011] Lustre: Failing over lustre-OST0000 [10166.062498] LustreError: 1295:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10166.065649] LustreError: 1295:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10166.076483] Lustre: server umount lustre-OST0000 complete [10167.306288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10167.313734] LustreError: 3989:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10168.818010] Lustre: 1903:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [10168.952460] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10168.961437] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10170.335729] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10170.661503] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10170.661729] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [10170.666804] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [10171.918840] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 22:51:27 (1746154287) [10184.884864] Lustre: Failing over lustre-OST0000 [10185.656215] Lustre: lustre-OST0000: Not available for connect from 192.168.203.46@tcp (stopping) [10185.688977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10185.697116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10186.934423] LustreError: 3456:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10186.937372] LustreError: 3456:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10186.943918] Lustre: server umount lustre-OST0000 complete [10188.968705] LustreError: 3989:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10189.321692] Lustre: 4186:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [10189.418951] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10189.425694] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10190.656283] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10190.722566] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10190.723566] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [10191.217605] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [10192.904759] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 22:51:48 (1746154308) [10217.129077] Lustre: Failing over lustre-OST0000 [10217.155443] LustreError: 6973:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10217.157757] LustreError: 6973:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10217.166830] Lustre: server umount lustre-OST0000 complete [10219.449917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10219.458788] LustreError: 11441:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10220.521629] Lustre: 7579:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [10220.688093] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10220.704092] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10220.734344] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10222.240651] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10222.241844] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [10222.955609] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [10224.655441] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 22:52:19 (1746154339) [10225.275553] Lustre: DEBUG MARKER: SKIP: sanity test_165e needs >= 2 MDTs [10226.113044] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 22:52:21 (1746154341) [10232.303528] Lustre: Failing over lustre-OST0000 [10232.339004] LustreError: 8591:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10232.342247] LustreError: 8591:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10232.351545] Lustre: server umount lustre-OST0000 complete [10235.674487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10235.680002] LustreError: 11441:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10238.592883] Lustre: 9674:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [10238.708674] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10238.718107] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10239.938799] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10240.478769] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10240.479899] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [10240.525581] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [10241.931610] Lustre: DEBUG MARKER: == sanity test 165g: ofd_access_log_reader --keepalive works ========================================================== 22:52:37 (1746154357) [10283.444397] Lustre: Failing over lustre-OST0000 [10283.477183] LustreError: 10670:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10283.482230] LustreError: 10670:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10283.492693] Lustre: server umount lustre-OST0000 complete [10285.563158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10285.567111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10285.575425] LustreError: 11441:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10285.582061] LustreError: 11441:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [10290.425403] Lustre: 11822:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [10290.539634] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10290.550004] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10292.129648] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [10292.168777] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [10292.170602] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [10292.475641] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [10294.083728] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 22:53:29 (1746154409) [10294.917614] Lustre: DEBUG MARKER: creating a 10 Mb file [10331.873641] Lustre: DEBUG MARKER: starting reads [10332.834624] Lustre: DEBUG MARKER: truncating the file [10333.783481] Lustre: DEBUG MARKER: killing dd [10334.449944] Lustre: DEBUG MARKER: removing the temporary file [10337.382719] Lustre: DEBUG MARKER: == sanity test 170a: test lctl df to handle corrupted log ========================================================== 22:54:12 (1746154452) [10341.219861] Lustre: DEBUG MARKER: == sanity test 170b: check filename encoding ============= 22:54:16 (1746154456) [10352.189793] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 22:54:27 (1746154467) [10357.872967] Lustre: DEBUG MARKER: == sanity test 172: manual device removal with lctl cleanup/detach ================================================================ 22:54:33 (1746154473) [10361.795378] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 22:54:37 (1746154477) [10362.356660] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [10363.073939] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 22:54:38 (1746154478) [10364.226888] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_module obdecho/obdecho [10372.274051] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 22:54:47 (1746154487) [10373.445392] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_module obdecho/obdecho [10373.510444] Lustre: Echo OBD driver; http://www.lustre.org/ [10382.165806] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 22:54:57 (1746154497) [10433.632591] Lustre: DEBUG MARKER: == sanity test 182a: Test parallel modify metadata operations from mdc ========================================================== 22:55:49 (1746154549) [10466.861278] Lustre: DEBUG MARKER: == sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 22:56:21 (1746154581) [10467.553112] Lustre: DEBUG MARKER: SKIP: sanity test_182b needs >= 2 MDTs [10468.442357] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 22:56:23 (1746154583) [10469.094872] Lustre: *** cfs_fail_loc=148, val=0*** [10472.975035] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 22:56:28 (1746154588) [10478.458424] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 22:56:33 (1746154593) [10482.127260] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 22:56:37 (1746154597) [10499.208191] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 22:56:54 (1746154614) [10503.930300] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 22:56:59 (1746154619) [10508.033037] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 22:57:03 (1746154623) [10510.923883] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 22:57:06 (1746154626) [10515.660224] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 22:57:10 (1746154630) [10521.170922] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 22:57:16 (1746154636) [10525.521944] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 22:57:20 (1746154640) [10528.517741] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 22:57:23 (1746154643) [10544.921147] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 22:57:40 (1746154660) [10547.182909] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 22:57:42 (1746154662) [10549.696828] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 22:57:45 (1746154665) [10552.023062] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 22:57:47 (1746154667) [10554.344590] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 22:57:49 (1746154669) [10556.753221] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 22:57:52 (1746154672) [10558.917573] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 22:57:54 (1746154674) [10561.002110] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 22:57:56 (1746154676) [10563.073092] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 22:57:58 (1746154678) [10566.644386] Lustre: lustre-MDD0000: changelog on [10566.646081] Lustre: Skipped 2 previous similar messages [10567.869108] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [10568.459678] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.16717 [10569.502247] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [10570.115860] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.25028 [10571.237620] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [10571.871170] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.6475 [10572.761893] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [10573.297115] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.6021 [10574.089802] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [10574.579294] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.14971 [10575.380617] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [10575.883862] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.1171 [10577.012525] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [10577.515823] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.958 [10578.545722] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [10579.101714] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.16465 [10579.969986] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [10580.581275] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.8904 [10581.918975] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [10582.553594] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.12657 [10583.729380] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [10584.386552] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.31601 [10588.993884] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [10589.620669] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg346-client.v [10590.673057] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [10591.320708] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg346-client.E [10592.370636] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [10592.953627] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.oleg346-client.v [10596.144699] Lustre: lustre-MDD0000: changelog off [10596.145869] Lustre: Skipped 1 previous similar message [10599.579367] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 22:58:34 (1746154714) [10603.834403] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 22:58:39 (1746154719) [10606.146163] Lustre: DEBUG MARKER: == sanity test 205d: verify the format of some stats files ========================================================== 22:58:41 (1746154721) [10612.371176] Lustre: DEBUG MARKER: == sanity test 205e: verify the output of lljobstat ====== 22:58:47 (1746154727) [10619.527136] Lustre: DEBUG MARKER: == sanity test 205f: verify qos_ost_weights YAML format == 22:58:54 (1746154734) [10622.876860] Lustre: DEBUG MARKER: == sanity test 205g: stress test for job_stats procfile == 22:58:58 (1746154738) [10717.650117] Lustre: DEBUG MARKER: == sanity test 205h: check jobid xattr is stored correctly ========================================================== 23:00:32 (1746154832) [10723.641234] Lustre: DEBUG MARKER: == sanity test 205i: check job_xattr parameter accepts and rejects values correctly ========================================================== 23:00:38 (1746154838) [10730.537864] Lustre: DEBUG MARKER: == sanity test 205k: Verify '?' operator on job stats ==== 23:00:45 (1746154845) [10735.234975] Lustre: DEBUG MARKER: == sanity test 205l: Verify job stats can scale ========== 23:00:50 (1746154850) [10806.769006] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 23:02:01 (1746154921) [10810.602409] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 23:02:05 (1746154925) [10814.125990] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 23:02:09 (1746154929) [10817.491260] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 23:02:12 (1746154932) [10824.746619] Lustre: Failing over lustre-MDT0000 [10824.947034] LustreError: 32116:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10824.949967] LustreError: 32116:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10825.106039] Lustre: server umount lustre-MDT0000 complete [10838.678092] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10838.896096] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10838.990257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10839.045583] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10840.340653] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [10841.894988] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [10841.990421] Lustre: 2339:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746154942/real 1746154942] req@ffff88011850f800 x1830972498642816/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746154958 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [10842.022318] Lustre: 2339:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10842.034449] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [10842.056780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000402:55358 to 0x240000402:55393) [10842.056950] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:48680 to 0x280000402:48705) [10843.950109] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10843.978099] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.203.146@tcp (at 0@lo) [10844.481809] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10847.027452] Lustre: 2339:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746154947/real 1746154947] req@ffff880076d84a80 x1830972498643200/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746154963 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [10847.039555] Lustre: 2339:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10848.469833] Lustre: Failing over lustre-MDT0000 [10848.644446] Lustre: server umount lustre-MDT0000 complete [10851.983631] Lustre: 2342:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746154952/real 1746154952] req@ffff880076d86300 x1830972498643456/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746154968 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [10851.993597] Lustre: 2342:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10861.281908] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10861.497865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10862.027397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000402:55358 to 0x240000402:55425) [10862.027409] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:48680 to 0x280000402:48737) [10862.717733] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [10865.443621] Lustre: 2342:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746154965/real 1746154965] req@ffff880076d85c00 x1830972498653952/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746154981 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [10865.461386] Lustre: 2342:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10865.983751] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10866.533817] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10870.403502] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 23:03:05 (1746154985) [10870.455384] Lustre: 2339:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746154970/real 1746154970] req@ffff880076d86300 x1830972498654208/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746154986 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [10870.466630] Lustre: 2339:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [10881.763517] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 23:03:17 (1746154997) [10886.729863] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 23:03:21 (1746155001) [10892.297503] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 23:03:27 (1746155007) [10905.359962] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 23:03:40 (1746155020) [10927.225366] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 23:04:02 (1746155042) [10931.128399] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 23:04:06 (1746155046) [10942.234797] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with embedded hyphen ('-') ========================================================== 23:04:17 (1746155057) [10947.406707] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 23:04:22 (1746155062) [10948.225252] Lustre: DEBUG MARKER: creating a 10 Mb file [10981.409611] Lustre: DEBUG MARKER: starting reads [10982.276309] Lustre: DEBUG MARKER: truncating the file [10983.545265] Lustre: DEBUG MARKER: killing dd [10984.552680] Lustre: DEBUG MARKER: removing the temporary file [10988.304979] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 23:05:03 (1746155103) [10992.597254] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 23:05:07 (1746155107) [10996.698567] Lustre: *** cfs_fail_loc=229, val=0*** [10998.959803] Lustre: *** cfs_fail_loc=229, val=0*** [10998.962101] Lustre: Skipped 1 previous similar message [11000.181676] Lustre: *** cfs_fail_loc=229, val=0*** [11000.183170] Lustre: Skipped 1 previous similar message [11010.337626] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 23:05:25 (1746155125) [11018.370940] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 23:05:33 (1746155133) [11021.728966] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 23:05:36 (1746155136) [11025.539362] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 23:05:40 (1746155140) [11030.664885] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 23:05:45 (1746155145) [11030.849016] Lustre: lustre-OST0001: Client a929ef20-6b20-47bc-ab1f-c730a17331f9 (at 192.168.203.46@tcp) reconnecting [11039.299163] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 23:05:54 (1746155154) [11041.599850] LustreError: 4761:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 sleeping for 3000ms [11044.611536] LustreError: 4761:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 awake [11048.695979] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 23:06:04 (1746155164) [11055.668154] Lustre: *** cfs_fail_loc=520, val=57344*** [11055.703779] LNet: *** cfs_fail_loc=e000, val=2147483648*** [11060.921380] Lustre: lustre-OST0000: Client a929ef20-6b20-47bc-ab1f-c730a17331f9 (at 192.168.203.46@tcp) reconnecting [11070.500593] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 23:06:25 (1746155185) [11072.748610] LustreError: 19412:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 sleeping for 22000ms [11093.002991] Lustre: lustre-OST0000: Client a929ef20-6b20-47bc-ab1f-c730a17331f9 (at 192.168.203.46@tcp) reconnecting [11094.516410] LustreError: 19412:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout interrupted [11094.519541] LustreError: 19412:0:(ldlm_lib.c:3490:target_bulk_io()) @@@ bulk READ failed: rc = -107 req@ffff880083625880 x1830972549105408/t0(0) o3->a929ef20-6b20-47bc-ab1f-c730a17331f9@192.168.203.46@tcp:0/0 lens 488/440 e 0 to 0 dl 1746155208 ref 1 fl Interpret:/200/0 rc 0/0 job:'dd.0' uid:0 gid:0 [11094.532408] Lustre: 19412:0:(service.c:2494:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/2s); client may timeout req@ffff880083625880 x1830972549105408/t0(0) o3->a929ef20-6b20-47bc-ab1f-c730a17331f9@192.168.203.46@tcp:0/0 lens 488/440 e 0 to 0 dl 1746155208 ref 1 fl Complete:/200/0 rc 0/0 job:'dd.0' uid:0 gid:0 [11098.532690] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [11099.452492] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [11100.447230] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 23:06:55 (1746155215) [11104.094222] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 23:06:59 (1746155219) [11104.809425] Lustre: DEBUG MARKER: SKIP: sanity test_226b needs >= 2 MDTs [11105.734109] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 23:07:00 (1746155220) [11106.581834] Lustre: DEBUG MARKER: SKIP: sanity test_226c needs >= 2 MDTs [11107.626557] Lustre: DEBUG MARKER: == sanity test 226d: verify fid2path with -n and -fn option ========================================================== 23:07:02 (1746155222) [11111.398863] Lustre: DEBUG MARKER: == sanity test 226e: Verify path2fid -0 option with newline and space ========================================================== 23:07:06 (1746155226) [11115.108828] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 23:07:10 (1746155230) [11118.631158] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 23:07:13 (1746155233) [11119.455230] Lustre: DEBUG MARKER: SKIP: sanity test_228a ldiskfs only test [11120.365527] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 23:07:15 (1746155235) [11121.220583] Lustre: DEBUG MARKER: SKIP: sanity test_228b ldiskfs only test [11121.972048] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 23:07:17 (1746155237) [11122.783480] Lustre: DEBUG MARKER: SKIP: sanity test_228c ldiskfs only test [11123.509000] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 23:07:18 (1746155238) [11126.461851] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 23:07:21 (1746155241) [11127.200695] Lustre: DEBUG MARKER: SKIP: sanity test_230a needs >= 2 MDTs [11128.022272] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 23:07:23 (1746155243) [11128.713256] Lustre: DEBUG MARKER: SKIP: sanity test_230b needs >= 2 MDTs [11129.611964] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 23:07:24 (1746155244) [11130.312938] Lustre: DEBUG MARKER: SKIP: sanity test_230c needs >= 2 MDTs [11131.130178] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [11131.956670] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 23:07:27 (1746155247) [11132.608810] Lustre: DEBUG MARKER: SKIP: sanity test_230e needs >= 2 MDTs [11133.350751] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 23:07:28 (1746155248) [11134.039283] Lustre: DEBUG MARKER: SKIP: sanity test_230f needs >= 2 MDTs [11134.833587] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 23:07:30 (1746155250) [11135.531232] Lustre: DEBUG MARKER: SKIP: sanity test_230g needs >= 2 MDTs [11136.426994] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 23:07:31 (1746155251) [11137.165631] Lustre: DEBUG MARKER: SKIP: sanity test_230h needs >= 2 MDTs [11138.072767] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 23:07:33 (1746155253) [11138.825207] Lustre: DEBUG MARKER: SKIP: sanity test_230i needs >= 2 MDTs [11139.642916] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 23:07:34 (1746155254) [11140.252307] Lustre: DEBUG MARKER: SKIP: sanity test_230j needs >= 2 MDTs [11140.969452] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 23:07:36 (1746155256) [11141.522393] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [11142.255749] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 23:07:37 (1746155257) [11142.854777] Lustre: DEBUG MARKER: SKIP: sanity test_230l needs >= 2 MDTs [11143.652205] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 23:07:38 (1746155258) [11144.341041] Lustre: DEBUG MARKER: SKIP: sanity test_230m needs >= 2 MDTs [11145.071953] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 23:07:40 (1746155260) [11145.697102] Lustre: DEBUG MARKER: SKIP: sanity test_230n needs >= 2 MDTs [11146.534235] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 23:07:41 (1746155261) [11147.264043] Lustre: DEBUG MARKER: SKIP: sanity test_230o needs >= 2 MDTs [11148.097565] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 23:07:43 (1746155263) [11148.747699] Lustre: DEBUG MARKER: SKIP: sanity test_230p needs >= 2 MDTs [11149.548243] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 23:07:44 (1746155264) [11150.244042] Lustre: DEBUG MARKER: SKIP: sanity test_230q needs >= 2 MDTs [11150.909600] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 23:07:46 (1746155266) [11151.487528] Lustre: DEBUG MARKER: SKIP: sanity test_230r needs >= 2 MDTs [11152.245073] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 23:07:47 (1746155267) [11157.238900] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 23:07:52 (1746155272) [11157.829766] Lustre: DEBUG MARKER: SKIP: sanity test_230t needs >= 2 MDTs [11158.512232] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 23:07:53 (1746155273) [11159.093996] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [11159.792887] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 23:07:55 (1746155275) [11160.440643] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [11161.103448] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 23:07:56 (1746155276) [11161.640104] Lustre: DEBUG MARKER: SKIP: sanity test_230w needs >= 2 MDTs [11162.287017] Lustre: DEBUG MARKER: == sanity test 230x: dir migration check space =========== 23:07:57 (1746155277) [11162.906158] Lustre: DEBUG MARKER: SKIP: sanity test_230x needs >= 2 MDTs [11163.657418] Lustre: DEBUG MARKER: == sanity test 230y: unlink dir with bad hash type ======= 23:07:59 (1746155279) [11164.343989] Lustre: DEBUG MARKER: SKIP: sanity test_230y needs >= 2 MDTs [11165.294133] Lustre: DEBUG MARKER: == sanity test 230z: resume dir migration with bad hash type ========================================================== 23:08:00 (1746155280) [11166.160039] Lustre: DEBUG MARKER: SKIP: sanity test_230z needs >= 2 MDTs [11167.150124] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 23:08:02 (1746155282) [11172.195187] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 23:08:07 (1746155287) [11197.496880] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 23:08:32 (1746155312) [11197.983099] Lustre: *** cfs_fail_loc=31c, val=0*** [11200.124195] Lustre: Failing over lustre-OST0000 [11200.196412] LustreError: 23724:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [11200.201823] LustreError: 23724:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [11200.215228] Lustre: server umount lustre-OST0000 complete [11200.516827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11200.520777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11200.525349] Lustre: Skipped 3 previous similar messages [11200.531049] LustreError: 3623:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11200.540350] LustreError: 3623:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [11203.387947] Lustre: 24362:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [11203.519114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11203.525791] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [11203.528840] Lustre: Skipped 1 previous similar message [11204.252852] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11204.256808] Lustre: Skipped 1 previous similar message [11204.836499] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11204.836710] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [11204.836711] Lustre: Skipped 3 previous similar messages [11204.849993] Lustre: Skipped 1 previous similar message [11205.516517] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [11209.350296] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 23:08:44 (1746155324) [11210.014301] Lustre: *** cfs_fail_loc=31c, val=0*** [11210.016591] LustreError: 11433:0:(ldlm_request.c:475:ldlm_cli_enqueue_local()) ### delayed lvb init failed (rc -12) ns: filter-lustre-OST0000_UUID lock: ffff880091b0be00/0xa15db558f6759928 lrc: 2/0,0 mode: --/PR res: [0x240000402:0xd968:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 11433 timeout: 0 lvb_type: 0 [11211.838792] Lustre: Failing over lustre-OST0000 [11211.925269] Lustre: server umount lustre-OST0000 complete [11213.515817] LustreError: 3624:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11215.190556] Lustre: 26168:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [11215.345153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11217.204755] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [11220.469669] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 23:08:55 (1746155335) [11223.357761] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 23:08:58 (1746155338) [11225.845555] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 23:09:01 (1746155341) [11228.721478] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 23:09:04 (1746155344) [11233.350631] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 23:09:08 (1746155348) [11236.315615] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 23:09:11 (1746155351) [11239.557277] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 23:09:14 (1746155354) [11296.783449] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 23:10:12 (1746155412) [11298.295128] Lustre: *** cfs_fail_loc=2100, val=0*** [11313.356219] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 23:10:28 (1746155428) [11330.828797] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 23:10:46 (1746155446) [11331.515253] Lustre: DEBUG MARKER: SKIP: sanity test_240 needs >= 2 MDTs [11332.492088] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 23:10:47 (1746155447) [11384.425272] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 23:11:39 (1746155499) [11409.759557] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 23:12:04 (1746155524) [11410.408923] Lustre: *** cfs_fail_loc=105, val=0*** [11414.156473] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 23:12:09 (1746155529) [11610.270770] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 23:15:25 (1746155725) [11682.302613] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 23:16:37 (1746155797) [11686.026759] Lustre: DEBUG MARKER: == sanity test 245a: check mdc connection flag/data: multiple modify RPCs ========================================================== 23:16:41 (1746155801) [11689.725650] Lustre: DEBUG MARKER: == sanity test 245b: check osp connection flag/data: multiple modify RPCs ========================================================== 23:16:44 (1746155804) [11690.769927] Lustre: DEBUG MARKER: SKIP: sanity test_245b needs >= 2 MDTs [11691.741125] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 23:16:46 (1746155806) [11696.376256] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 23:16:51 (1746155811) [11700.636604] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 23:16:55 (1746155815) [11706.099272] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 23:17:01 (1746155821) [11711.489883] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 23:17:06 (1746155826) [11715.406378] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 23:17:10 (1746155830) [11716.252731] Lustre: DEBUG MARKER: SKIP: sanity test_247f needs >= 2 MDTs [11717.217019] Lustre: DEBUG MARKER: == sanity test 247g: striped directory submount revalidate ROOT from cache ========================================================== 23:17:12 (1746155832) [11718.135332] Lustre: DEBUG MARKER: SKIP: sanity test_247g needs > 1 MDTs [11719.165922] Lustre: DEBUG MARKER: == sanity test 247h: remote directory submount revalidate ROOT from cache ========================================================== 23:17:14 (1746155834) [11720.112492] Lustre: DEBUG MARKER: SKIP: sanity test_247h needs > 1 MDTs [11721.114309] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 23:17:16 (1746155836) [11796.441862] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 23:18:31 (1746155911) [11853.161980] Lustre: DEBUG MARKER: == sanity test 248c: verify whole file read behavior ===== 23:19:28 (1746155968) [11866.208847] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 23:19:41 (1746155981) [11869.575943] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 23:19:44 (1746155984) [11870.364643] Lustre: DEBUG MARKER: SKIP: sanity test_250 no 16TB file size limit on ZFS [11871.305128] Lustre: DEBUG MARKER: == sanity test 251a: Handling short read and write correctly ========================================================== 23:19:46 (1746155986) [11875.478971] Lustre: DEBUG MARKER: == sanity test 251b: short read restore offset correctly ========================================================== 23:19:50 (1746155990) [11883.166896] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 23:19:58 (1746155998) [11884.202956] Lustre: DEBUG MARKER: SKIP: sanity test_252 ldiskfs only test [11885.236175] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 23:20:00 (1746156000) [11886.277277] Lustre: DEBUG MARKER: SKIP: sanity test_253 need >= 2.13.57 and ldiskfs for fallocate [11887.147487] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 23:20:02 (1746156002) [11888.958801] Lustre: lustre-MDD0000: changelog on [11893.232137] Lustre: lustre-MDD0000: changelog off [11894.393703] Lustre: DEBUG MARKER: SKIP: sanity test_255a skipping excluded test 255a (base 255) [11895.120853] Lustre: DEBUG MARKER: SKIP: sanity test_255b skipping excluded test 255b (base 255) [11895.981391] Lustre: DEBUG MARKER: SKIP: sanity test_255c skipping excluded test 255c (base 255) [11896.817175] Lustre: DEBUG MARKER: SKIP: sanity test_256 skipping excluded test 256 [11897.476797] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 23:20:12 (1746156012) [11898.200097] Lustre: *** cfs_fail_loc=161, val=0*** [11899.063142] Lustre: Failing over lustre-MDT0000 [11899.199381] LustreError: 28983:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11899.203398] LustreError: 28983:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11899.337014] Lustre: server umount lustre-MDT0000 complete [11902.531405] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11902.715065] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11902.720846] Lustre: Skipped 2 previous similar messages [11902.792903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11902.839695] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11902.842387] Lustre: Skipped 1 previous similar message [11904.063780] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [11905.867974] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [11907.452702] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [11907.455471] Lustre: Skipped 1 previous similar message [11907.556031] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [11907.562705] Lustre: Skipped 1 previous similar message [11907.580597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000402:58178 to 0x240000402:58209) [11907.580766] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:51439 to 0x280000402:51457) [11907.771439] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.203.146@tcp (at 0@lo) [11907.776797] Lustre: Skipped 1 previous similar message [11913.934707] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 23:20:29 (1746156029) [11917.737912] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 23:20:32 (1746156032) [11917.744664] Lustre: 2341:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746156017/real 1746156017] req@ffff880073ff8380 x1830972499662080/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746156033 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [11917.766052] Lustre: 2341:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [11921.745575] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 23:20:36 (1746156036) [11922.543232] Lustre: DEBUG MARKER: SKIP: sanity test_259 ldiskfs only test [11923.570908] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 23:20:38 (1746156038) [11927.513390] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 23:20:42 (1746156042) [11936.220179] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 23:20:51 (1746156051) [11940.298597] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 23:20:55 (1746156055) [11944.181865] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 23:20:59 (1746156059) [11947.816159] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 23:21:03 (1746156063) [11952.449776] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 23:21:07 (1746156067) [11955.022957] Lustre: Increasing provided stripe size to a minimum value 64 [11961.119005] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 23:21:16 (1746156076) [11965.088373] Lustre: *** cfs_fail_loc=168, val=40*** [11966.055708] Lustre: *** cfs_fail_loc=168, val=20*** [11966.057602] Lustre: Skipped 2 previous similar messages [11968.028331] Lustre: *** cfs_fail_loc=168, val=15*** [11968.029992] Lustre: Skipped 5 previous similar messages [11970.044331] Lustre: *** cfs_fail_loc=168, val=55*** [11970.048214] Lustre: Skipped 6 previous similar messages [11975.008516] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 23:21:30 (1746156090) [11980.956662] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 23:21:36 (1746156096) [11984.799474] Lustre: DEBUG MARKER: == sanity test 270j: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 23:21:39 (1746156099) [11989.407767] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 23:21:44 (1746156104) [11993.224988] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 23:21:48 (1746156108) [11996.638147] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 23:21:51 (1746156111) [12000.415415] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 23:21:55 (1746156115) [12073.988650] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 23:23:09 (1746156189) [12078.031700] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 23:23:13 (1746156193) [12082.303171] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 23:23:17 (1746156197) [12086.115725] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 23:23:21 (1746156201) [12090.251281] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 23:23:25 (1746156205) [12095.979626] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 23:23:31 (1746156211) [12101.139946] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 23:23:36 (1746156216) [12106.895197] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 23:23:42 (1746156222) [12112.211344] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 23:23:47 (1746156227) [12116.462496] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 23:23:51 (1746156231) [12121.255687] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 23:23:56 (1746156236) [12122.605684] LustreError: 29710:0:(mdt_io.c:571:mdt_commitrw_write()) cfs_fail_timeout id 16b sleeping for 2000ms [12124.620423] LustreError: 29710:0:(mdt_io.c:571:mdt_commitrw_write()) cfs_fail_timeout id 16b awake [12127.839190] Lustre: DEBUG MARKER: == sanity test 273c: race writeback and object destroy === 23:24:02 (1746156242) [12129.044626] LustreError: 4763:0:(ofd_io.c:1283:ofd_commitrw_write()) cfs_fail_timeout id 1e1 sleeping for 2000ms [12131.050543] LustreError: 4763:0:(ofd_io.c:1283:ofd_commitrw_write()) cfs_fail_timeout id 1e1 awake [12134.048721] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 23:24:09 (1746156249) [12139.744872] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 23:24:15 (1746156255) [12140.580623] Lustre: Failing over lustre-OST0000 [12140.581646] LustreError: 13488:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [12140.637489] LustreError: 13483:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [12140.640541] LustreError: 13483:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12140.648217] Lustre: server umount lustre-OST0000 complete [12142.824183] LustreError: 4757:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12143.157806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12143.163654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12144.483476] Lustre: 14652:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [12144.807588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12144.816627] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [12146.156081] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [12146.820538] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [12146.823831] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [12146.823832] Lustre: Skipped 1 previous similar message [12147.375756] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12149.180447] Lustre: Failing over lustre-OST0000 [12149.181708] LustreError: 16057:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [12149.181710] LustreError: 16057:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [12149.301096] Lustre: server umount lustre-OST0000 complete [12149.723051] LustreError: 11435:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12149.730151] LustreError: 11435:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [12153.820772] Lustre: 17250:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [12153.876416] LustreError: 17288:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [12153.887181] LustreError: 17288:0:(obd_class.h:479:obd_check_dev()) Skipped 6 previous similar messages [12154.045329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12156.793151] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12159.297057] Lustre: Failing over lustre-OST0000 [12159.383949] LustreError: 18640:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [12159.394515] LustreError: 18640:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12159.414264] Lustre: server umount lustre-OST0000 complete [12160.538048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12160.542332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12160.561467] Lustre: Skipped 1 previous similar message [12160.565490] LustreError: 11435:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12160.576270] LustreError: 11435:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [12164.305038] Lustre: 19785:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [12164.489738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12164.501053] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [12164.508057] Lustre: Skipped 1 previous similar message [12165.715672] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [12165.721699] Lustre: Skipped 1 previous similar message [12166.579612] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [12166.582631] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [12166.582633] Lustre: Skipped 1 previous similar message [12166.597202] Lustre: Skipped 1 previous similar message [12167.206513] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12168.904708] Lustre: Failing over lustre-OST0000 [12168.906628] LustreError: 21156:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [12168.987646] Lustre: server umount lustre-OST0000 complete [12169.497939] LustreError: 11435:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12169.507617] LustreError: 11435:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [12172.257233] Lustre: 22318:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [12172.426909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12174.716890] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12176.402475] Lustre: Failing over lustre-OST0000 [12176.405203] LustreError: 23662:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [12176.408189] LustreError: 23662:0:(obd_class.h:479:obd_check_dev()) Skipped 2 previous similar messages [12176.450358] Lustre: server umount lustre-OST0000 complete [12180.250475] Lustre: 24851:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [12182.284784] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12183.822522] Lustre: Failing over lustre-OST0000 [12183.892251] Lustre: server umount lustre-OST0000 complete [12186.998123] LustreError: 27480:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [12187.001741] LustreError: 27480:0:(obd_class.h:479:obd_check_dev()) Skipped 2 previous similar messages [12187.113155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12187.118590] Lustre: Skipped 1 previous similar message [12189.814552] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12191.829833] Lustre: Failing over lustre-OST0000 [12191.883457] LustreError: 28810:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [12191.886574] LustreError: 28810:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [12191.901439] Lustre: server umount lustre-OST0000 complete [12192.091855] LustreError: 3622:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12192.103652] LustreError: 3622:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [12196.230785] Lustre: 30026:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [12196.235255] Lustre: 30026:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message [12197.922171] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [12197.925681] Lustre: Skipped 3 previous similar messages [12199.150126] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12201.395578] Lustre: Failing over lustre-OST0000 [12201.405375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12201.405377] Lustre: Skipped 4 previous similar messages [12201.585236] Lustre: server umount lustre-OST0000 complete [12205.925762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12205.931540] Lustre: Skipped 1 previous similar message [12205.946275] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [12205.948415] Lustre: Skipped 4 previous similar messages [12208.024852] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [12208.027771] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [12208.027773] Lustre: Skipped 4 previous similar messages [12208.035567] Lustre: Skipped 4 previous similar messages [12208.386618] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12210.151056] Lustre: Failing over lustre-OST0000 [12210.151182] LustreError: 1507:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [12210.151184] LustreError: 1507:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [12210.238006] Lustre: server umount lustre-OST0000 complete [12214.125062] Lustre: 2712:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [12214.129601] Lustre: 2712:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message [12217.598990] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12219.508402] Lustre: Failing over lustre-OST0000 [12219.679696] Lustre: server umount lustre-OST0000 complete [12221.576934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12226.247122] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12228.238065] Lustre: Failing over lustre-OST0000 [12228.296910] Lustre: server umount lustre-OST0000 complete [12229.001937] LustreError: 11446:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12229.013736] LustreError: 11446:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [12234.880832] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12236.428537] Lustre: Failing over lustre-OST0000 [12236.470574] Lustre: server umount lustre-OST0000 complete [12239.355440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12239.360027] Lustre: Skipped 3 previous similar messages [12241.353048] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12242.913487] Lustre: Failing over lustre-OST0000 [12242.915859] LustreError: 11548:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [12242.921563] LustreError: 11548:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [12242.997460] Lustre: server umount lustre-OST0000 complete [12246.588318] Lustre: 12686:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.146@tcp: no remote llog for lustre-sptlrpc, check MGS config [12246.597455] Lustre: 12686:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 3 previous similar messages [12249.031184] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12250.959098] Lustre: Failing over lustre-OST0000 [12251.028963] Lustre: server umount lustre-OST0000 complete [12257.147813] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12258.823461] Lustre: Failing over lustre-OST0000 [12258.852684] LustreError: 16511:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [12258.858276] LustreError: 16511:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [12258.875491] Lustre: server umount lustre-OST0000 complete [12263.029524] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [12263.032969] Lustre: Skipped 7 previous similar messages [12264.563748] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12266.229475] Lustre: Failing over lustre-OST0000 [12266.292841] Lustre: server umount lustre-OST0000 complete [12267.400498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12267.405730] Lustre: Skipped 7 previous similar messages [12271.946999] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12273.458646] Lustre: Failing over lustre-OST0000 [12273.515026] Lustre: server umount lustre-OST0000 complete [12277.047832] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [12277.052028] Lustre: Skipped 8 previous similar messages [12278.249928] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [12278.253698] Lustre: Skipped 8 previous similar messages [12278.256350] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.146@tcp (at 0@lo) [12278.260498] Lustre: Skipped 8 previous similar messages [12279.319856] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12280.977253] Lustre: Failing over lustre-OST0000 [12281.047410] Lustre: server umount lustre-OST0000 complete [12287.614918] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12289.603051] Lustre: Failing over lustre-OST0000 [12289.697430] Lustre: server umount lustre-OST0000 complete [12293.083490] LustreError: 11435:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12293.096815] LustreError: 11435:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 13 previous similar messages [12296.174246] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12298.263954] Lustre: Failing over lustre-OST0000 [12298.332000] Lustre: server umount lustre-OST0000 complete [12304.694147] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12310.115362] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 23:27:05 (1746156425) [12313.936923] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 23:27:09 (1746156429) [12314.705243] Lustre: DEBUG MARKER: SKIP: sanity test_278 needs >= 2 MDTs [12315.850104] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 23:27:10 (1746156430) [12317.193672] LustreError: 29693:0:(llog_osd.c:1480:llog_osd_open()) cfs_race id 15e sleeping [12319.061571] Lustre: Failing over lustre-MDT0000 [12319.238458] LustreError: 32520:0:(tgt_mount.c:128:server_deregister_mount()) cfs_fail_race id 15e waking [12319.243726] LustreError: 29693:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e awake: rc=2952 [12320.254463] LustreError: 32520:0:(tgt_mount.c:128:server_deregister_mount()) cfs_fail_race id 15e waking [12320.419590] Lustre: server umount lustre-MDT0000 complete [12324.705049] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12324.848580] LustreError: 727:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [12325.982968] LustreError: 726:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [12326.995677] LustreError: 726:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [12329.027966] LustreError: 726:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [12329.030913] LustreError: 726:0:(llog_osd.c:1480:llog_osd_open()) Skipped 1 previous similar message [12333.053236] LustreError: 725:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [12333.056226] LustreError: 725:0:(llog_osd.c:1480:llog_osd_open()) Skipped 3 previous similar messages [12335.111597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12335.114510] Lustre: Skipped 8 previous similar messages [12335.227132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000402:58241 to 0x240000402:58273) [12335.227999] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:51482 to 0x280000402:51521) [12336.677861] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug all all [12338.064438] Lustre: 2342:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746156438/real 1746156438] req@ffff880085e58a80 x1830972500115584/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746156454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [12338.076911] Lustre: 2342:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12341.718374] LustreError: 726:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [12341.723369] LustreError: 726:0:(llog_osd.c:1480:llog_osd_open()) Skipped 5 previous similar messages [12346.597264] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 23:27:41 (1746156461) [12347.286659] Lustre: DEBUG MARKER: SKIP: sanity test_300a needs >= 2 MDTs [12348.302032] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 23:27:43 (1746156463) [12349.151145] Lustre: DEBUG MARKER: SKIP: sanity test_300b needs >= 2 MDTs [12350.136679] Lustre: DEBUG MARKER: == sanity test 300c: chown [12351.012030] Lustre: DEBUG MARKER: SKIP: sanity test_300c needs >= 2 MDTs [12352.124565] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 23:27:47 (1746156467) [12352.913699] Lustre: DEBUG MARKER: SKIP: sanity test_300d needs >= 2 MDTs [12353.697147] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ========================================================== 23:27:49 (1746156469) [12354.500998] Lustre: DEBUG MARKER: SKIP: sanity test_300e needs >= 2 MDTs [12355.359726] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ========================================================== 23:27:50 (1746156470) [12356.165401] Lustre: DEBUG MARKER: SKIP: sanity test_300f needs >= 2 MDTs [12357.174619] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ========================================================== 23:27:52 (1746156472) [12358.033514] Lustre: DEBUG MARKER: SKIP: sanity test_300g needs >= 2 MDTs [12358.950507] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory ========================================================== 23:27:54 (1746156474) [12359.852156] Lustre: DEBUG MARKER: SKIP: sanity test_300h needs >= 2 MDTs [12360.884063] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory ========================================================== 23:27:56 (1746156476) [12361.530259] Lustre: DEBUG MARKER: SKIP: sanity test_300i needs >= 2 MDTs [12362.365107] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ============ 23:27:57 (1746156477) [12363.114474] Lustre: DEBUG MARKER: SKIP: sanity test_300j needs >= 2 MDTs [12364.054231] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ======== 23:27:59 (1746156479) [12364.836981] Lustre: DEBUG MARKER: SKIP: sanity test_300k needs >= 2 MDTs [12365.830713] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ========================================================== 23:28:01 (1746156481) [12366.519956] Lustre: DEBUG MARKER: SKIP: sanity test_300l needs >= 2 MDTs [12367.654502] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS ========================================================== 23:28:02 (1746156482) [12372.020802] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA ========================================================== 23:28:07 (1746156487) [12372.931109] Lustre: DEBUG MARKER: SKIP: sanity test_300n needs >= 2 MDTs [12373.945096] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [12374.904872] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================================== 23:28:10 (1746156490) [12375.815825] Lustre: DEBUG MARKER: SKIP: sanity test_300p needs >= 2 MDTs [12376.841280] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ========================================================== 23:28:11 (1746156491) [12377.750464] Lustre: DEBUG MARKER: SKIP: sanity test_300q needs >= 2 MDTs [12378.794761] Lustre: DEBUG MARKER: == sanity test 300r: test -1 striped directory =========== 23:28:13 (1746156493) [12379.733649] Lustre: DEBUG MARKER: SKIP: sanity test_300r needs >= 2 MDTs [12380.807582] Lustre: DEBUG MARKER: == sanity test 300s: test lfs mkdir -c without -i ======== 23:28:15 (1746156495) [12381.766607] Lustre: DEBUG MARKER: SKIP: sanity test_300s needs >= 2 MDTs [12382.674655] Lustre: DEBUG MARKER: == sanity test 300t: test max_mdt_stripecount ============ 23:28:17 (1746156497) [12383.532908] Lustre: DEBUG MARKER: SKIP: sanity test_300t needs at least 2 MDTs [12385.648210] Lustre: DEBUG MARKER: == sanity test 300ua: basic overstriped dir sanity test == 23:28:20 (1746156500) [12386.417528] Lustre: DEBUG MARKER: SKIP: sanity test_300ua needs >= 2 MDTs [12387.429210] Lustre: DEBUG MARKER: == sanity test 300ub: test MDT overstriping interface [12388.092472] Lustre: DEBUG MARKER: SKIP: sanity test_300ub needs >= 2 MDTs [12388.864515] Lustre: DEBUG MARKER: == sanity test 300uc: test MDT overstriping as default [12389.510082] Lustre: DEBUG MARKER: SKIP: sanity test_300uc needs >= 2 MDTs [12390.211768] Lustre: DEBUG MARKER: == sanity test 300ud: dir split ========================== 23:28:25 (1746156505) [12390.788527] Lustre: DEBUG MARKER: SKIP: sanity test_300ud needs >= 2 MDTs [12391.486697] Lustre: DEBUG MARKER: == sanity test 300ue: dir merge ========================== 23:28:26 (1746156506) [12392.004534] Lustre: DEBUG MARKER: SKIP: sanity test_300ue needs >= 2 MDTs [12392.646826] Lustre: DEBUG MARKER: == sanity test 300uf: migrate with too many local locks == 23:28:28 (1746156508) [12393.219854] Lustre: DEBUG MARKER: SKIP: sanity test_300uf needs >= 2 MDTs [12393.834990] Lustre: DEBUG MARKER: == sanity test 300ug: migrate overstriped dirs =========== 23:28:29 (1746156509) [12394.420084] Lustre: DEBUG MARKER: SKIP: sanity test_300ug needs >= 2 MDTs [12395.119051] Lustre: DEBUG MARKER: == sanity test 300uh: overstripe tunable max_stripes_per_mdt ========================================================== 23:28:30 (1746156510) [12395.734825] Lustre: DEBUG MARKER: SKIP: sanity test_300uh needs >= 2 MDTs [12396.429304] Lustre: DEBUG MARKER: == sanity test 300ui: overstripe is not supported on one MDT system ========================================================== 23:28:31 (1746156511) [12399.392644] Lustre: DEBUG MARKER: == sanity test 300uj: overstriped dir with -C -N sanity test ========================================================== 23:28:34 (1746156514) [12400.223082] Lustre: DEBUG MARKER: SKIP: sanity test_300uj needs >= 2 MDTs [12401.649921] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ============= 23:28:36 (1746156516) [12402.267598] Lustre: DEBUG MARKER: SKIP: sanity test_310a needs >= 4 MDTs [12403.032712] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open ========================================================== 23:28:38 (1746156518) [12403.655592] Lustre: DEBUG MARKER: SKIP: sanity test_310b needs >= 4 MDTs [12404.361962] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ========================================================== 23:28:39 (1746156519) [12404.959514] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [12405.675826] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ========================================================== 23:28:41 (1746156521) [12450.604629] Lustre: DEBUG MARKER: SKIP: sanity test_312 skipping ALWAYS excluded test 312 [12451.285434] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ========================================================== 23:29:26 (1746156566) [12451.736385] Lustre: *** cfs_fail_loc=720, val=0*** [12451.738488] LustreError: 4763:0:(ofd_objects.c:1014:ofd_object_punch()) lustre-OST0000: failed to stop transaction: rc = -5 [12454.627953] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ========================================================== 23:29:30 (1746156570) [12455.745712] Lustre: *** cfs_fail_loc=720, val=0*** [12455.745767] LustreError: 11446:0:(ofd_objects.c:1081:ofd_destroy()) lustre-OST0001 failed to stop transaction: -5 [12455.745800] LustreError: 11446:0:(ofd_dev.c:1765:ofd_destroy_hdl()) lustre-OST0001: error destroying object [0x280000402:0xc943:0x0]: -5 [12455.755316] Lustre: Skipped 1 previous similar message [12460.475057] LustreError: 774:0:(osp_sync.c:1155:osp_sync_process_committed()) lustre-OST0001-osc-MDT0000: can't cancel record: rc = -2 [12475.903035] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ============= 23:29:51 (1746156591) [12482.535791] Lustre: DEBUG MARKER: == sanity test 316: lfs migrate of file with large_xattr enabled ========================================================== 23:29:57 (1746156597) [12483.139812] Lustre: DEBUG MARKER: SKIP: sanity test_316 needs >= 2 MDTs [12483.867465] Lustre: DEBUG MARKER: == sanity test 317: Verify blocks get correctly update after truncate ========================================================== 23:29:59 (1746156599) [12484.619430] Lustre: DEBUG MARKER: SKIP: sanity test_317 LU-10370: no implementation for ZFS [12485.458161] Lustre: DEBUG MARKER: == sanity test 318: Verify async readahead tunables ====== 23:30:00 (1746156600) [12488.332623] Lustre: DEBUG MARKER: == sanity test 319: lost lease lock on migrate error ===== 23:30:03 (1746156603) [12488.996548] Lustre: DEBUG MARKER: SKIP: sanity test_319 needs >= 2 MDTs [12489.715246] Lustre: DEBUG MARKER: == sanity test 350: force NID mismatch path to be exercised ========================================================== 23:30:05 (1746156605) [12490.406964] LNet: *** cfs_fail_loc=e001, val=100*** [12490.409771] LNet: 2332:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-192.168.203.46@tcp matched 1830972500290944 with NID mismatch 192.168.203.46@tcp accepted (same peer ffff8800654d6800) [12490.932657] LNet: *** cfs_fail_loc=e001, val=100*** [12490.935689] LNet: Skipped 23 previous similar messages [12490.938993] LNet: 2332:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-192.168.203.46@tcp matched 1830972500294016 with NID mismatch 192.168.203.46@tcp accepted (same peer ffff8800654d6800) [12490.945837] LNet: 2332:0:(lib-ptl.c:160:lnet_try_match_md()) Skipped 23 previous similar messages [12491.936197] LNet: *** cfs_fail_loc=e001, val=100*** [12491.937999] LNet: Skipped 23 previous similar messages [12491.940153] LNet: 2332:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-192.168.203.46@tcp matched 1830972500297088 with NID mismatch 192.168.203.46@tcp accepted (same peer ffff8800654d6800) [12491.948020] LNet: 2332:0:(lib-ptl.c:160:lnet_try_match_md()) Skipped 23 previous similar messages [12688.608288] Lustre: DEBUG MARKER: == sanity test 360: ldiskfs unlink in a separate thread == 23:33:23 (1746156803) [12689.355939] Lustre: DEBUG MARKER: SKIP: sanity test_360 ldiskfs only test [12690.176613] Lustre: DEBUG MARKER: == sanity test 398a: direct IO should cancel lock otherwise lockless ========================================================== 23:33:25 (1746156805) [12693.487537] Lustre: DEBUG MARKER: == sanity test 398b: DIO and buffer IO race ============== 23:33:28 (1746156808) [12876.096025] Lustre: DEBUG MARKER: == sanity test 398c: run fio to test AIO ================= 23:36:31 (1746156991) [13062.811965] Lustre: DEBUG MARKER: == sanity test 398d: run aiocp to verify block size > stripe size ========================================================== 23:39:38 (1746157178) [13078.182873] Lustre: DEBUG MARKER: == sanity test 398e: O_Direct open cleared by fcntl doesn't cause hang ========================================================== 23:39:53 (1746157193) [13081.996921] Lustre: DEBUG MARKER: == sanity test 398f: verify aio handles ll_direct_rw_pages errors correctly ========================================================== 23:39:57 (1746157197) [13088.377154] Lustre: DEBUG MARKER: == sanity test 398g: verify parallel dio async RPC submission ========================================================== 23:40:03 (1746157203) [13089.222029] LustreError: 12565:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [13089.226459] LustreError: 12565:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [13091.227679] LustreError: 12568:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [13091.232174] LustreError: 12568:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [13091.640975] LustreError: 12565:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [13091.644456] LustreError: 12565:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 3 previous similar messages [13093.649428] LustreError: 12565:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [13093.653584] LustreError: 12565:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 4 previous similar messages [13093.717994] LustreError: 12565:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [13095.721631] LustreError: 12565:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [13095.818702] LustreError: 12565:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [13097.825587] LustreError: 12565:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [13099.954738] LustreError: 12565:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [13099.961044] LustreError: 12565:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [13101.972623] LustreError: 12565:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [13101.975578] LustreError: 12565:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [13108.273708] LustreError: 4762:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [13108.283100] LustreError: 4762:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 3 previous similar messages [13108.986450] LustreError: 4762:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout interrupted [13111.638331] Lustre: DEBUG MARKER: == sanity test 398h: verify correctness of read [13123.140750] Lustre: DEBUG MARKER: == sanity test 398i: verify parallel dio handles ll_direct_rw_pages errors correctly ========================================================== 23:40:38 (1746157238) [13128.040268] Lustre: DEBUG MARKER: == sanity test 398j: test parallel dio where stripe size > rpc_size ========================================================== 23:40:43 (1746157243) [13139.995646] Lustre: DEBUG MARKER: == sanity test 398k: test enospc on first stripe ========= 23:40:55 (1746157255) [13162.479786] Lustre: DEBUG MARKER: SKIP: sanity test_398k 7508992 > 600000 skipping out-of-space test on OST0 [13163.301494] Lustre: DEBUG MARKER: == sanity test 398l: test enospc on intermediate stripe/RPC ========================================================== 23:41:18 (1746157278) [13173.675206] Lustre: DEBUG MARKER: SKIP: sanity test_398l 7487488 > 600000 skipping out-of-space test on OST0 [13190.954722] Lustre: DEBUG MARKER: == sanity test 398m: test RPC failures with parallel dio ========================================================== 23:41:46 (1746157306) [13191.605520] Lustre: *** cfs_fail_loc=20e, val=1*** [13192.644966] Lustre: *** cfs_fail_loc=20e, val=1*** [13192.650132] Lustre: Skipped 7 previous similar messages [13194.691709] Lustre: *** cfs_fail_loc=20e, val=1*** [13194.697363] Lustre: Skipped 3 previous similar messages [13197.700836] Lustre: *** cfs_fail_loc=20e, val=1*** [13201.739466] Lustre: *** cfs_fail_loc=20e, val=1*** [13201.747302] Lustre: Skipped 6 previous similar messages [13212.784065] Lustre: *** cfs_fail_loc=20e, val=1*** [13212.789231] Lustre: Skipped 7 previous similar messages [13236.204956] Lustre: *** cfs_fail_loc=20e, val=1*** [13236.215343] Lustre: Skipped 11 previous similar messages [13269.247264] Lustre: *** cfs_fail_loc=20f, val=1*** [13269.251005] Lustre: Skipped 35 previous similar messages [13340.361573] Lustre: *** cfs_fail_loc=20e, val=2*** [13340.364007] Lustre: Skipped 55 previous similar messages [13421.167447] Lustre: DEBUG MARKER: == sanity test 398n: test append with parallel DIO ======= 23:45:36 (1746157536) [13432.362342] Lustre: DEBUG MARKER: == sanity test 398o: right kms with DIO ================== 23:45:47 (1746157547) [13433.147978] Lustre: DEBUG MARKER: sanity test_398o: @@@@@@ FAIL: bad KMS [13439.678206] Lustre: DEBUG MARKER: == sanity test 398p: race aio with buffered i/o ========== 23:45:54 (1746157554) [13479.366331] Lustre: DEBUG MARKER: == sanity test 398q: race dio with buffered i/o ========== 23:46:34 (1746157594) [13676.397294] Lustre: DEBUG MARKER: == sanity test 398r: i/o error on file read ============== 23:49:51 (1746157791) [13676.969672] Lustre: *** cfs_fail_loc=20f, val=0*** [13676.971390] Lustre: Skipped 55 previous similar messages [13736.560413] Lustre: DEBUG MARKER: == sanity test 398s: i/o error on mirror file read ======= 23:50:51 (1746157851) [13737.844528] Lustre: DEBUG MARKER: sanity test_398s: @@@@@@ FAIL: mirror resync failed