[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 321322429 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.470376] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.473104] pid_max: default: 32768 minimum: 301 [ 0.474381] Security Framework initialized [ 0.475246] SELinux: Initializing. [ 0.476544] SELinux: Starting in permissive mode [ 0.478032] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.483188] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.485690] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.488610] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.493459] Initializing cgroup subsys memory [ 0.496071] Initializing cgroup subsys devices [ 0.499283] Initializing cgroup subsys freezer [ 0.502638] Initializing cgroup subsys net_cls [ 0.506088] Initializing cgroup subsys blkio [ 0.508877] Initializing cgroup subsys perf_event [ 0.510489] Initializing cgroup subsys hugetlb [ 0.512230] Initializing cgroup subsys pids [ 0.514055] Initializing cgroup subsys net_prio [ 0.515620] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.519094] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.521086] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.523118] tlb_flushall_shift: 6 [ 0.524493] FEATURE SPEC_CTRL Present [ 0.526033] FEATURE IBPB_SUPPORT Present [ 0.527514] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.530103] Spectre V2 : Vulnerable [ 0.531291] Speculative Store Bypass: Vulnerable [ 0.534360] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.542640] ACPI: Core revision 20130517 [ 0.545891] ACPI: All ACPI Tables successfully acquired [ 0.549563] ftrace: allocating 30294 entries in 119 pages [ 0.605132] Enabling x2apic [ 0.606412] Enabled x2apic [ 0.608283] Switched APIC routing to physical x2apic. [ 0.612022] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.614491] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.618690] TSC deadline timer enabled [ 0.618950] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.621875] ... version: 2 [ 0.623400] ... bit width: 48 [ 0.624791] ... generic registers: 4 [ 0.626195] ... value mask: 0000ffffffffffff [ 0.628157] ... max period: 00007fffffffffff [ 0.629869] ... fixed-purpose events: 3 [ 0.631375] ... event mask: 000000070000000f [ 0.633450] KVM setup paravirtual spinlock [ 0.639612] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.643160] KVM setup async PF for cpu 1 [ 0.644701] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.648023] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.637455] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.651777] KVM setup async PF for cpu 2 [ 0.651786] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.655198] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.659203] Brought up 4 CPUs [ 0.659207] KVM setup async PF for cpu 3 [ 0.659216] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.662922] smpboot: Max logical packages: 1 [ 0.664140] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.669000] devtmpfs: initialized [ 0.670530] x86/mm: Memory block size: 128MB [ 0.675482] EVM: security.selinux [ 0.676939] EVM: security.ima [ 0.679019] EVM: security.capability [ 0.685362] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.690887] NET: Registered protocol family 16 [ 0.694726] cpuidle: using governor haltpoll [ 0.698347] ACPI: bus type PCI registered [ 0.701231] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.706322] PCI: Using configuration type 1 for base access [ 0.710416] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.722739] ACPI: Added _OSI(Module Device) [ 0.725877] ACPI: Added _OSI(Processor Device) [ 0.729165] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.732596] ACPI: Added _OSI(Processor Aggregator Device) [ 0.736519] ACPI: Added _OSI(Linux-Dell-Video) [ 0.740958] ACPI: EC: Look up EC in DSDT [ 0.744541] ACPI: Interpreter enabled [ 0.747378] ACPI: (supports S0 S3 S4 S5) [ 0.750266] ACPI: Using IOAPIC for interrupt routing [ 0.754004] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.759888] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.771249] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.773858] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.776136] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.782778] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.787339] acpiphp: Slot [2] registered [ 0.788475] acpiphp: Slot [3] registered [ 0.789392] acpiphp: Slot [4] registered [ 0.790756] acpiphp: Slot [5] registered [ 0.792452] acpiphp: Slot [6] registered [ 0.795526] acpiphp: Slot [7] registered [ 0.797201] acpiphp: Slot [8] registered [ 0.798834] acpiphp: Slot [9] registered [ 0.800463] acpiphp: Slot [10] registered [ 0.801866] acpiphp: Slot [11] registered [ 0.803471] acpiphp: Slot [12] registered [ 0.804842] acpiphp: Slot [13] registered [ 0.806332] acpiphp: Slot [14] registered [ 0.808031] acpiphp: Slot [15] registered [ 0.809621] acpiphp: Slot [16] registered [ 0.811249] acpiphp: Slot [17] registered [ 0.812713] acpiphp: Slot [18] registered [ 0.814305] acpiphp: Slot [19] registered [ 0.815775] acpiphp: Slot [20] registered [ 0.817484] acpiphp: Slot [21] registered [ 0.818919] acpiphp: Slot [22] registered [ 0.820452] acpiphp: Slot [23] registered [ 0.822049] acpiphp: Slot [24] registered [ 0.823623] acpiphp: Slot [25] registered [ 0.824995] acpiphp: Slot [26] registered [ 0.826341] acpiphp: Slot [27] registered [ 0.827897] acpiphp: Slot [28] registered [ 0.829381] acpiphp: Slot [29] registered [ 0.830873] acpiphp: Slot [30] registered [ 0.832349] acpiphp: Slot [31] registered [ 0.833721] PCI host bridge to bus 0000:00 [ 0.835187] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.837407] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.839588] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.842070] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.846650] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.849357] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.851418] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.852256] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.853198] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.859588] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.862040] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.863744] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.866791] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.869141] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.871854] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.872460] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.874918] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.877798] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.879756] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.888184] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.890957] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.894697] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.899404] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.903614] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.921387] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.925371] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.928460] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.932254] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.944101] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.947908] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.950916] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.956804] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.965791] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.974484] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.977491] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.980122] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.993139] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 1.004968] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.007711] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.010340] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.022141] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 1.028499] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.033453] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.037827] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.047225] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 1.053890] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.056061] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.058529] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.060683] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.062707] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.065581] vgaarb: loaded [ 1.066837] SCSI subsystem initialized [ 1.067951] ACPI: bus type USB registered [ 1.069354] usbcore: registered new interface driver usbfs [ 1.071240] usbcore: registered new interface driver hub [ 1.072955] usbcore: registered new device driver usb [ 1.075037] PCI: Using ACPI for IRQ routing [ 1.076387] PCI: pci_cache_line_size set to 64 bytes [ 1.076660] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.076666] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.076668] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.077585] NetLabel: Initializing [ 1.078793] NetLabel: domain hash size = 128 [ 1.080334] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.082083] NetLabel: unlabeled traffic allowed by default [ 1.084104] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.085593] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.090996] amd_nb: Cannot enumerate AMD northbridges [ 1.093092] Switched to clocksource kvm-clock [ 1.117967] pnp: PnP ACPI init [ 1.120203] ACPI: bus type PNP registered [ 1.123213] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.123359] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.123439] pnp 00:02: [dma 2] [ 1.123488] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.123615] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.123734] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.123858] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.124485] pnp: PnP ACPI: found 6 devices [ 1.127221] ACPI: bus type PNP unregistered [ 1.146954] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.146963] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.146965] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.146968] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.146971] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.147071] NET: Registered protocol family 2 [ 1.150744] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.156073] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.162152] TCP: Hash tables configured (established 32768 bind 32768) [ 1.166602] TCP: reno registered [ 1.167605] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.169539] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.171941] NET: Registered protocol family 1 [ 1.174639] RPC: Registered named UNIX socket transport module. [ 1.176189] RPC: Registered udp transport module. [ 1.177846] RPC: Registered tcp transport module. [ 1.179308] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.181632] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.184064] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.186379] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.188378] PCI: CLS 0 bytes, default 64 [ 1.189200] Unpacking initramfs... [ 2.653945] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.660205] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.664751] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.671539] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.677131] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.680906] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.684781] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.691707] cryptomgr_test (51) used greatest stack depth: 14128 bytes left [ 2.692486] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.692617] Initialise system trusted keyring [ 2.723691] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.725951] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.734036] zpool: loaded [ 2.735242] zbud: loaded [ 2.736649] VFS: Disk quotas dquot_6.6.0 [ 2.738136] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.741507] NFS: Registering the id_resolver key type [ 2.742853] Key type id_resolver registered [ 2.744271] Key type id_legacy registered [ 2.745657] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.748393] Key type big_key registered [ 2.749893] SELinux: Registering netfilter hooks [ 2.754067] cryptomgr_test (57) used greatest stack depth: 14048 bytes left [ 2.760441] cryptomgr_test (62) used greatest stack depth: 13984 bytes left [ 2.763064] cryptomgr_test (60) used greatest stack depth: 13968 bytes left [ 2.764860] NET: Registered protocol family 38 [ 2.764873] Key type asymmetric registered [ 2.764877] Asymmetric key parser 'x509' registered [ 2.765018] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.765212] io scheduler noop registered [ 2.765217] io scheduler deadline registered (default) [ 2.765282] io scheduler cfq registered [ 2.765287] io scheduler mq-deadline registered [ 2.765294] io scheduler kyber registered [ 2.767719] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.767730] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.786603] intel_idle: does not run on family 6 model 62 [ 2.786752] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.789051] ACPI: Power Button [PWRF] [ 2.790701] GHES: HEST is not enabled! [ 2.850372] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.913019] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 3.059481] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 3.131121] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 3.255892] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.283636] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 3.311237] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 3.314695] Non-volatile memory driver v1.3 [ 3.315796] Linux agpgart interface v0.103 [ 3.317324] crash memory driver: version 1.1 [ 3.318669] nbd: registered device at major 43 [ 3.324207] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 3.324248] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 3.324275] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 3.324297] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 3.324331] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 3.330509] virtio_blk virtio1: [vda] 115440 512-byte logical blocks (59.1 MB/56.3 MiB) [ 3.335799] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 3.335838] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 3.335871] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 3.335905] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 3.335949] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 3.343563] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.350077] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 3.350146] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 3.350205] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 3.350252] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 3.350292] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 3.355674] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.361448] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 3.361493] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 3.361537] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 3.361586] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 3.361627] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 3.368026] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.373788] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 3.373816] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 3.373847] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 3.373870] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 3.373894] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 3.380053] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.385483] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 3.385527] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 3.385570] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 3.385608] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 3.385645] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 3.392257] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.397634] rdac: device handler registered [ 3.399397] hp_sw: device handler registered [ 3.400858] emc: device handler registered [ 3.402362] libphy: Fixed MDIO Bus: probed [ 3.407146] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.409714] ehci-pci: EHCI PCI platform driver [ 3.411451] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.413564] ohci-pci: OHCI PCI platform driver [ 3.415179] uhci_hcd: USB Universal Host Controller Interface driver [ 3.417354] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.421130] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.422870] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.425190] mousedev: PS/2 mouse device common for all mice [ 3.428015] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.431777] rtc_cmos 00:05: RTC can wake from S4 [ 3.434948] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.438477] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.443291] hidraw: raw HID events driver (C) Jiri Kosina [ 3.445213] usbcore: registered new interface driver usbhid [ 3.447108] usbhid: USB HID core driver [ 3.448611] drop_monitor: Initializing network drop monitor service [ 3.450601] Netfilter messages via NETLINK v0.30. [ 3.452200] TCP: cubic registered [ 3.453232] Initializing XFRM netlink socket [ 3.455440] NET: Registered protocol family 10 [ 3.458003] NET: Registered protocol family 17 [ 3.459839] Key type dns_resolver registered [ 3.461897] mce: Using 10 MCE banks [ 3.463599] Loading compiled-in X.509 certificates [ 3.466193] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.469655] registered taskstats version 1 [ 3.474075] modprobe (71) used greatest stack depth: 13456 bytes left [ 3.479526] Key type trusted registered [ 3.486923] modprobe (79) used greatest stack depth: 13440 bytes left [ 3.489557] Key type encrypted registered [ 3.491458] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.495854] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.500226] rtc_cmos 00:05: setting system clock to 2025-05-01 23:09:29 UTC (1746140969) [ 3.503398] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.506346] Write protecting the kernel read-only data: 12288k [ 3.509172] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.511737] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.520430] random: systemd: uninitialized urandom read (16 bytes read) [ 3.524398] random: systemd: uninitialized urandom read (16 bytes read) [ 3.526773] random: systemd: uninitialized urandom read (16 bytes read) [ 3.531046] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.537825] systemd[1]: Detected virtualization kvm. [ 3.539579] systemd[1]: Detected architecture x86-64. [ 3.541068] systemd[1]: Running in initial RAM disk. [ 3.546062] systemd[1]: No hostname configured. [ 3.547781] systemd[1]: Set hostname to . [ 3.549572] random: systemd: uninitialized urandom read (16 bytes read) [ 3.551922] systemd[1]: Initializing machine ID from random generator. [ 3.565578] ln (88) used greatest stack depth: 13008 bytes left [ 3.603068] random: systemd: uninitialized urandom read (16 bytes read) [ 3.605292] random: systemd: uninitialized urandom read (16 bytes read) [ 3.607684] random: systemd: uninitialized urandom read (16 bytes read) [ 3.610466] random: systemd: uninitialized urandom read (16 bytes read) [ 3.614187] random: systemd: uninitialized urandom read (16 bytes read) [ 3.616964] random: systemd: uninitialized urandom read (16 bytes read) [ 3.628120] systemd[1]: Reached target Local File Systems. [ 3.633441] systemd[1]: Created slice Root Slice. [ 3.637450] systemd[1]: Listening on Journal Socket. [ 3.643221] systemd[1]: Listening on udev Control Socket. [ 3.647343] systemd[1]: Reached target Swap. [ 3.651039] systemd[1]: Reached target Timers. [ 3.655717] systemd[1]: Created slice System Slice. [ 3.662923] systemd[1]: Starting Setup Virtual Console... [ 3.666910] systemd[1]: Reached target Slices. [ 3.673061] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.681924] systemd[1]: Starting Journal Service... [ 3.686927] systemd[1]: Starting dracut cmdline hook... [ 3.691145] tsc: Refined TSC clocksource calibration: 2399.977 MHz [ 3.691251] systemd[1]: Listening on udev Kernel Socket. [ 3.698059] systemd[1]: Reached target Sockets. [ 3.704476] systemd[1]: Starting Load Kernel Modules... [ 3.710004] systemd[1]: Started Setup Virtual Console. [ 3.717244] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 3.726031] systemd[1]: Started Load Kernel Modules. [ 3.731329] systemd[1]: Started Journal Service. [ 3.843050] random: fast init done [ 4.263812] libata version 3.00 loaded. [ 4.270641] ata_piix 0000:00:01.1: version 2.13 [ 4.276602] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 4.278056] scsi host0: ata_piix [ 4.278307] scsi host1: ata_piix [ 4.278389] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.278390] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 4.521504] ip (343) used greatest stack depth: 12336 bytes left [ 7.582301] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 8.066759] systemd-journald[104]: Received SIGTERM from PID 1 (systemd). [ 8.310361] SELinux: Disabled at runtime. [ 8.312050] SELinux: Unregistering netfilter hooks [ 8.377800] ip_tables: (C) 2000-2006 Netfilter Core Team [ 8.381640] systemd[1]: Inserted module 'ip_tables' [ 8.821070] systemd-journald[565]: Received request to flush runtime journal from PID 1 [ 8.961981] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 8.989571] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 9.026428] cryptd: max_cpu_qlen set to 1000 [ 9.051303] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 9.055499] AVX version of gcm_enc/dec engaged. [ 9.057615] AES CTR mode by8 optimization enabled [ 9.092365] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 9.122717] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 9.126215] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 9.247715] EDAC MC: Ver: 3.0.0 [ 9.258788] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 9.258804] EDAC sbridge: Ver: 1.1.2 [ 11.430446] mount.nfs (770) used greatest stack depth: 10704 bytes left [ 23.840879] device-mapper: uevent: version 1.0.3 [ 23.843042] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 28.769494] libcfs: loading out-of-tree module taints kernel. [ 28.771827] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 28.803799] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 34.060120] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 34.314983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 34.319992] alg: No test for adler32 (adler32-zlib) [ 35.240421] Lustre: Lustre: Build Version: 2.16.54_75_gccae094 [ 35.445136] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 35.450565] LNet: Accept secure, port 988 [ 37.013265] Key type lgssc registered [ 37.382576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 40.698541] icp: module license 'CDDL' taints kernel. [ 40.700607] Disabling lock debugging due to kernel taint [ 42.289922] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 45.876980] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 50.882334] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 53.345935] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 53.356291] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 57.127365] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 57.134621] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 62.379399] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 67.892997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 67.941994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 67.955928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 69.132723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 69.164692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 69.218996] Lustre: lustre-MDT0000: new disk, initializing [ 69.320983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 69.336642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 69.403378] mount.lustre (7199) used greatest stack depth: 9728 bytes left [ 70.954821] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 77.940264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.973014] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 78.020111] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 78.026238] Lustre: 8360:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 78.041985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 78.049169] Lustre: Skipped 1 previous similar message [ 78.107470] Lustre: lustre-MDT0001: new disk, initializing [ 78.137869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 78.151439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 78.161077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 79.832207] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 82.831862] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 87.720017] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 87.728177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.776118] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 87.783180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 87.898482] Lustre: 9561:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 87.915681] Lustre: lustre-OST0000: new disk, initializing [ 87.935119] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 87.967624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 90.478392] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 91.325469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 91.330463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 91.380864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 95.642808] random: crng init done [ 98.245766] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 98.252854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.278586] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 98.285984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 98.327399] Lustre: 10904:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 98.338662] Lustre: lustre-OST0001: new disk, initializing [ 98.341554] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 98.374986] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 101.053727] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.493266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 107.501845] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 107.530401] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 108.984344] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 113.658269] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 121.654968] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing check_logdir /tmp/testlogs/ [ 123.576650] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing yml_node [ 126.182118] Lustre: DEBUG MARKER: Client: 2.16.54.75 [ 127.501509] Lustre: DEBUG MARKER: MDS: 2.16.54.75 [ 129.036617] Lustre: DEBUG MARKER: OSS: 2.16.54.75 [ 129.853405] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Thu May 1 19:11:35 EDT 2025 [ 139.287428] Lustre: DEBUG MARKER: - need mds1 <= 2.14.55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 140.269118] Lustre: DEBUG MARKER: - need mds1 < v2_14_55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 141.093462] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42c 42b 118c 118d 407 119i 851 411b [ 141.931005] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 [ 142.751670] Lustre: DEBUG MARKER: === sanity: start setup 19:11:48 (1746141108) === [ 144.589246] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing check_config_client /mnt/lustre [ 155.516530] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 157.392424] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 158.887962] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.079678] Lustre: DEBUG MARKER: === sanity: finish setup 19:12:07 (1746141127) === [ 165.608967] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 19:12:11 (1746141131) [ 167.702871] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 168.554140] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 19:12:14 (1746141134) [ 172.161154] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 19:12:17 (1746141137) [ 274.456986] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 19:14:00 (1746141240) [ 277.776644] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 19:14:03 (1746141243) [ 278.438591] Lustre: *** cfs_fail_loc=15b, val=0*** [ 278.440226] Lustre: *** cfs_fail_loc=15b, val=0*** [ 281.923687] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 19:14:07 (1746141247) [ 285.314836] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 19:14:11 (1746141251) [ 285.827446] Lustre: *** cfs_fail_loc=19a, val=0*** [ 286.624805] Lustre: *** cfs_fail_loc=19a, val=0*** [ 286.627581] Lustre: Skipped 1 previous similar message [ 287.911502] Lustre: *** cfs_fail_loc=19a, val=0*** [ 287.915811] Lustre: Skipped 2 previous similar messages [ 290.127225] Lustre: *** cfs_fail_loc=19a, val=0*** [ 290.130601] Lustre: Skipped 4 previous similar messages [ 294.231478] Lustre: *** cfs_fail_loc=19a, val=0*** [ 294.236066] Lustre: Skipped 9 previous similar messages [ 302.729769] Lustre: *** cfs_fail_loc=19a, val=0*** [ 302.732134] Lustre: Skipped 17 previous similar messages [ 318.952339] Lustre: *** cfs_fail_loc=19a, val=0*** [ 318.955915] Lustre: Skipped 30 previous similar messages [ 341.687737] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 19:15:07 (1746141307) [ 342.112940] Lustre: *** cfs_fail_loc=188, val=0*** [ 343.609449] Lustre: *** cfs_fail_loc=189, val=0*** [ 347.933393] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 348.728637] Lustre: DEBUG MARKER: == sanity test 60j: llog_reader reports corruptions ====== 19:15:14 (1746141314) [ 351.658928] Lustre: lustre-MDD0000: changelog on [ 353.069430] Lustre: lustre-MDD0001: changelog on [ 357.639497] Lustre: DEBUG MARKER: SKIP: sanity test_60j path oi.1/0x1:0xb:0x0 is not in 'O/1/d/' format [ 359.217147] Lustre: lustre-MDD0001: changelog off [ 360.596034] Lustre: lustre-MDD0000: changelog off [ 361.898189] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 19:15:27 (1746141327) [ 365.061552] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 19:15:30 (1746141330) [ 368.401800] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 19:15:34 (1746141334) [ 432.467398] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 19:16:38 (1746141398) [ 439.643760] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 19:16:45 (1746141405) [ 443.181880] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 443.870455] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 19:16:49 (1746141409) [ 447.612019] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 19:16:53 (1746141413) [ 468.954740] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 19:17:14 (1746141434) [ 470.346664] Lustre: *** cfs_fail_loc=725, val=0*** [ 472.516335] Lustre: *** cfs_fail_loc=725, val=0*** [ 475.975628] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 19:17:21 (1746141441) [ 480.276383] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 19:17:26 (1746141446) [ 550.883376] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 19:18:36 (1746141516) [ 562.484475] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 19:18:48 (1746141528) [ 565.280373] Lustre: *** cfs_fail_loc=513, val=17*** [ 565.282248] LustreError: 17259:0:(service.c:2266:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1830961547292672 [ 565.924926] Lustre: Failing over lustre-OST0000 [ 565.945349] LustreError: 2367:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 566.008194] Lustre: server umount lustre-OST0000 complete [ 567.204528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 567.205161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.218481] LustreError: Skipped 1 previous similar message [ 572.397993] LustreError: 12142:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.406448] LustreError: 12142:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 573.268630] LustreError: 17259:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.274762] LustreError: 17259:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 577.404341] LustreError: 17259:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.065307] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 579.072900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 579.131119] Lustre: 2905:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 579.191320] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 580.824111] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 580.853861] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [ 581.888709] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.101@tcp (at 0@lo) [ 581.889016] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 584.305613] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 584.990548] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 589.724813] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 19:19:15 (1746141555) [ 592.431812] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 19:19:18 (1746141558) [ 594.959438] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 19:19:20 (1746141560) [ 597.983748] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 19:19:23 (1746141563) [ 600.504376] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 19:19:26 (1746141566) [ 602.770602] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 19:19:28 (1746141568) [ 605.053329] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 19:19:31 (1746141571) [ 608.001790] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 19:19:33 (1746141573) [ 610.873893] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 19:19:36 (1746141576) [ 614.442292] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 19:19:40 (1746141580) [ 618.268069] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 19:19:44 (1746141584) [ 619.109381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.115042] Lustre: Skipped 1 previous similar message [ 619.118209] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 619.122341] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.101@tcp (at 0@lo) [ 619.128807] Lustre: Skipped 1 previous similar message [ 619.920959] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 619.924177] Lustre: Skipped 1 previous similar message [ 620.278807] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.283841] Lustre: Skipped 2 previous similar messages [ 620.286978] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.203.101@tcp (at 0@lo) [ 620.296024] Lustre: Skipped 2 previous similar messages [ 629.052734] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 636.466234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.477274] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 636.481571] Lustre: Skipped 1 previous similar message [ 636.485439] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 636.491346] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.101@tcp (at 0@lo) [ 638.098132] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 638.199536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 639.670024] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 639.762700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 648.570475] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 656.481137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 656.487634] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 656.492286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 656.500926] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.101@tcp (at 0@lo) [ 658.067388] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 658.147385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 659.346438] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 659.434043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 668.259264] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 674.120830] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 674.127274] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 674.132395] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 674.137752] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.203.101@tcp (at 0@lo) [ 675.081214] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 675.140751] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 676.100966] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 676.156529] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 682.347103] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 689.161212] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 689.167261] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 690.571613] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 690.659533] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 691.837252] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 691.913287] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 694.567083] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 19:21:00 (1746141660) [ 697.179826] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 19:21:03 (1746141663) [ 699.560151] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 19:21:05 (1746141665) [ 713.594838] Lustre: DEBUG MARKER: == sanity test 65o: pool inheritance for mdt component === 19:21:19 (1746141679) [ 729.466985] Lustre: DEBUG MARKER: == sanity test 65p: setstripe with yaml file and huge number ========================================================== 19:21:35 (1746141695) [ 732.122471] Lustre: DEBUG MARKER: == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 19:21:38 (1746141698) [ 734.439757] Lustre: DEBUG MARKER: == sanity test 65r: prevent all-zero offsets ============= 19:21:40 (1746141700) [ 737.213992] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 19:21:43 (1746141703) [ 742.672788] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 19:21:48 (1746141708) [ 743.406134] Lustre: *** cfs_fail_loc=217, val=0*** [ 743.409532] Lustre: Skipped 1 previous similar message [ 744.456226] Lustre: *** cfs_fail_loc=217, val=0*** [ 747.628796] Lustre: DEBUG MARKER: == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 19:21:53 (1746141713) [ 753.966410] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 754.795777] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 19:22:00 (1746141720) [ 757.822609] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 19:22:03 (1746141723) [ 761.207122] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 19:22:07 (1746141727) [ 790.890646] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 19:22:36 (1746141756) [ 794.284590] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 19:22:40 (1746141760) [ 798.109746] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 19:22:43 (1746141763) [ 801.522387] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 19:22:47 (1746141767) [ 845.532887] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 19:23:31 (1746141811) [ 872.192117] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 19:23:58 (1746141838) [ 875.275736] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 19:24:01 (1746141841) [ 875.562461] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.1@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [0-4194303]: client csum d2c0f6ef, server csum d2c0f6ee [ 877.386261] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 878.611377] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.1@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [0-4194303], client returned csum d9c4a5b8 (type 1), server csum 593b5aca (type 1) [ 879.876229] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 881.102172] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.1@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [0-4194303], client returned csum 4e3af8aa (type 2), server csum 8f2df9cd (type 2) [ 882.111157] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 883.325485] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.1@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [0-4194303], client returned csum 2fa490d9 (type 4), server csum d2c0f6ee (type 4) [ 884.496249] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 885.673337] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.1@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [0-4194303], client returned csum 187b013b (type 10), server csum 17f70117 (type 10) [ 886.767795] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 889.071885] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 890.296338] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.1@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [0-4194303], client returned csum f0eac0b9 (type 40), server csum f3aec185 (type 40) [ 890.311245] LustreError: Skipped 1 previous similar message [ 891.761067] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 893.874185] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 896.995140] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 19:24:22 (1746141862) [ 900.218589] Lustre: 27937:0:(tgt_handler.c:1935:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000407:0xc1f:0x0]:[0-1048575]-bac502c-ad3a6826 [ 900.231317] LustreError: dumping log to /tmp/lustre-log.1746141866.27937 [ 900.292560] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.1@tcp inode [0x200000407:0xc1f:0x0] object 0x280000401:3839 extent [0-1048575], client returned csum bac502c (type 4), server csum ad3a6826 (type 4) [ 900.304216] LustreError: Skipped 1 previous similar message [ 910.006134] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 19:24:35 (1746141875) [ 910.149010] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.1@tcp inode [0x200000407:0xc21:0x0] object 0x2c0000401:3831 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 915.721617] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 19:24:41 (1746141881) [ 916.603671] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 916.721993] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.1@tcp inode [0x200000407:0xc22:0x0] object 0x280000401:3840 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 919.998464] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.1@tcp inode [0x200000407:0xc22:0x0] object 0x280000401:3840 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 920.010398] LustreError: Skipped 3 previous similar messages [ 927.238637] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.1@tcp inode [0x200000407:0xc22:0x0] object 0x280000401:3840 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 927.249856] LustreError: Skipped 4 previous similar messages [ 938.097413] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.1@tcp inode [0x200000407:0xc22:0x0] object 0x280000401:3840 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 938.106557] LustreError: Skipped 2 previous similar messages [ 962.058775] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.1@tcp inode [0x200000407:0xc22:0x0] object 0x280000401:3840 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 962.077182] LustreError: Skipped 5 previous similar messages [ 972.919838] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 994.122834] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.1@tcp inode [0x200000407:0xc22:0x0] object 0x280000401:3840 extent [4194304-8388607]: client csum 7d33b9a0, server csum 7d33b99f [ 994.138502] LustreError: Skipped 15 previous similar messages [ 1028.927602] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1065.240849] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.1@tcp inode [0x200000407:0xc22:0x0] object 0x280000401:3840 extent [4194304-8388607]: client csum de2bf73f, server csum de2bf73e [ 1065.251755] LustreError: Skipped 25 previous similar messages [ 1084.904486] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1141.188623] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1198.466917] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.1@tcp inode [0x200000407:0xc22:0x0] object 0x280000401:3840 extent [4194304-8388607]: client csum fec05011, server csum fec05010 [ 1198.482196] LustreError: Skipped 48 previous similar messages [ 1199.301070] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1258.298941] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1314.477878] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1317.950726] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 19:31:23 (1746142283) [ 1318.672648] Lustre: *** cfs_fail_loc=21a, val=0*** [ 1321.158267] Lustre: *** cfs_fail_loc=21b, val=0*** [ 1326.430355] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 19:31:32 (1746142292) [ 1326.917197] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1328.139414] Lustre: Modifying parameter lustre.osc.lustre*.checksums in log params [ 1331.302303] Lustre: Disabling parameter lustre.osc.lustre*.checksums in log params [ 1335.740244] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1337.685449] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 19:31:43 (1746142303) [ 1338.510896] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 1339.389917] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1341.174117] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid 50 [ 1343.883940] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid in IDLE state after 2 sec [ 1345.297740] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid 50 [ 1345.858427] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid in FULL state after 0 sec [ 1346.537223] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1348.282813] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid 50 [ 1359.390930] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid in IDLE state after 10 sec [ 1360.758376] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid 50 [ 1361.421672] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid in FULL state after 0 sec [ 1362.070609] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1364.011271] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid 50 [ 1374.043424] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid in IDLE state after 9 sec [ 1375.479300] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid 50 [ 1376.331818] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid in FULL state after 0 sec [ 1377.116315] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1378.785763] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid 50 [ 1389.719242] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid in IDLE state after 10 sec [ 1391.197867] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid 50 [ 1391.993194] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid in FULL state after 0 sec [ 1393.063706] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1394.973993] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid 50 [ 1404.024525] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid in IDLE state after 8 sec [ 1405.688614] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid 50 [ 1406.518528] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid in FULL state after 0 sec [ 1407.279693] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1408.929008] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid 50 [ 1419.250595] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid in IDLE state after 9 sec [ 1421.293582] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid 50 [ 1421.995462] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid in FULL state after 0 sec [ 1422.603429] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1424.332735] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid 50 [ 1434.342532] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid in IDLE state after 9 sec [ 1435.968282] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid 50 [ 1436.704118] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012ab33000.ost_server_uuid in FULL state after 0 sec [ 1440.027862] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1440.978630] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 19:33:26 (1746142406) [ 1444.295790] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 19:33:30 (1746142410) [ 1445.253747] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1445.953990] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1446.794840] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1447.758957] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1450.722942] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1451.431992] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 19:33:37 (1746142417) [ 1456.623894] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 19:33:42 (1746142422) [ 1460.625432] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check =========== 19:33:46 (1746142426) [ 1475.161461] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 19:34:00 (1746142440) [ 1479.566564] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 19:34:05 (1746142445) [ 1480.274558] Lustre: *** cfs_fail_loc=228, val=0*** [ 1483.030476] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 19:34:08 (1746142448) [ 1483.635745] Lustre: *** cfs_fail_loc=228, val=0*** [ 1486.758456] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 19:34:12 (1746142452) [ 1501.837329] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ========================================================== 19:34:27 (1746142467) [ 1505.571443] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 19:34:31 (1746142471) [ 1607.176845] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 19:36:12 (1746142572) [ 1622.071212] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 19:36:27 (1746142587) [ 1658.965060] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 19:37:04 (1746142624) [ 1838.341403] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 19:40:03 (1746142803) [ 1882.212186] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 19:40:47 (1746142847) [ 1886.585363] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 19:40:52 (1746142852) [ 1932.713844] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 19:41:38 (1746142898) [ 1941.023311] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 19:41:46 (1746142906) [ 1945.476733] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 19:41:51 (1746142911) [ 1973.129237] Lustre: DEBUG MARKER: == sanity test 101m: read ahead for small file and last stripe of the file ========================================================== 19:42:19 (1746142939) [ 1980.211698] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 19:42:25 (1746142945) [ 1983.922519] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 19:42:29 (1746142949) [ 1990.939940] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 19:42:36 (1746142956) [ 1995.367926] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 19:42:41 (1746142961) [ 2015.039772] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 19:43:00 (1746142980) [ 2034.821772] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 19:43:20 (1746143000) [ 2035.444575] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 2036.105141] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 2036.747905] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 2037.419186] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2039.854273] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 19:43:25 (1746143005) [ 2040.973190] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2041.673754] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 2042.437975] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 2043.291192] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2043.987851] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2046.299281] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 19:43:32 (1746143012) [ 2048.927424] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 19:43:34 (1746143014) [ 2063.666139] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 19:43:49 (1746143029) [ 2066.514806] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 19:43:52 (1746143032) [ 2069.476015] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 19:43:55 (1746143035) [ 2072.286033] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 19:43:58 (1746143038) [ 2075.394078] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 19:44:01 (1746143041) [ 2077.951621] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 19:44:03 (1746143043) [ 2080.465295] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 19:44:06 (1746143046) [ 2083.561078] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 19:44:09 (1746143049) [ 2086.374886] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 19:44:12 (1746143052) [ 2089.960200] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 19:44:15 (1746143055) [ 2211.858738] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 19:46:17 (1746143177) [ 2303.645768] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 19:47:49 (1746143269) [ 2307.203866] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 19:47:52 (1746143272) [ 2390.933738] Lustre: lustre-MDT0000: Client 3a13c24d-0a8e-4058-9fa0-7c498aa0a5f4 (at 192.168.203.1@tcp) reconnecting [ 2493.313496] Lustre: lustre-MDT0000: Client 3a13c24d-0a8e-4058-9fa0-7c498aa0a5f4 (at 192.168.203.1@tcp) reconnecting [ 2767.588941] Lustre: lustre-MDT0000: Client 3a13c24d-0a8e-4058-9fa0-7c498aa0a5f4 (at 192.168.203.1@tcp) reconnecting [ 2803.881051] Lustre: lustre-MDT0000: Client 3a13c24d-0a8e-4058-9fa0-7c498aa0a5f4 (at 192.168.203.1@tcp) reconnecting [ 2884.039736] Lustre: lustre-MDT0000: Client 3a13c24d-0a8e-4058-9fa0-7c498aa0a5f4 (at 192.168.203.1@tcp) reconnecting [ 2897.563148] Lustre: 7222:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 30372 > trans_max 3200 [ 2897.568065] Lustre: 7222:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 2897.571515] Lustre: 7222:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28148/0 [ 2897.575405] Lustre: 7222:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 2897.579035] Lustre: 7222:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 2897.582701] Lustre: 7222:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 2897.588697] CPU: 3 PID: 7222 Comm: mdt00_001 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 2897.593700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 2897.597023] Call Trace: [ 2897.598442] [] dump_stack+0x19/0x1b [ 2897.600954] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 2897.604886] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 2897.608573] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 2897.612732] [] lod_trans_start+0x7f/0x2e0 [lod] [ 2897.616928] [] mdd_trans_start+0x14/0x20 [mdd] [ 2897.621653] [] mdd_unlink+0x617/0xea0 [mdd] [ 2897.624450] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 2897.627123] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 2897.629640] [] mdt_reint_rec+0x87/0x240 [mdt] [ 2897.631893] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 2897.634522] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 2897.637631] [] mdt_reint+0x67/0x150 [mdt] [ 2897.640210] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 2897.644615] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 2897.649525] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 2897.651665] [] ? put_prev_entity+0x31/0x400 [ 2897.653641] [] ? do_raw_spin_unlock+0x49/0x90 [ 2897.655533] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 2897.657033] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 2897.659484] [] kthread+0xe4/0xf0 [ 2897.661304] [] ? kthread_create_on_node+0x140/0x140 [ 2897.663265] [] ret_from_fork_nospec_begin+0x7/0x21 [ 2897.665687] [] ? kthread_create_on_node+0x140/0x140 [ 2900.523907] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 19:57:46 (1746143866) [ 2902.243913] Lustre: lustre-MDD0000: changelog on [ 2903.556458] Lustre: lustre-MDD0001: changelog on [ 2907.584353] Lustre: lustre-MDD0001: changelog off [ 2909.058439] Lustre: lustre-MDD0000: changelog off [ 2910.516994] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 19:57:56 (1746143876) [ 2912.518021] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88007f941800.ost_server_uuid 50 [ 2913.174066] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88007f941800.ost_server_uuid in FULL state after 0 sec [ 2916.850013] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 19:58:02 (1746143882) [ 2919.909460] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 19:58:05 (1746143885) [ 2920.668885] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [ 2921.684134] Lustre: DEBUG MARKER: == sanity test 104d: runas -u 500 -g 500 lctl dl test ==== 19:58:07 (1746143887) [ 2925.289928] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 19:58:11 (1746143891) [ 2928.655027] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 19:58:14 (1746143894) [ 2931.526810] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 19:58:17 (1746143897) [ 2934.230384] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 19:58:20 (1746143900) [ 2948.241973] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 19:58:33 (1746143913) [ 2951.556371] Lustre: DEBUG MARKER: == sanity test 105f: Enqueue same range flocks =========== 19:58:37 (1746143917) [ 2955.419892] Lustre: DEBUG MARKER: == sanity test 105g: ldlm_lock_debug stack test ========== 19:58:41 (1746143921) [ 2960.728299] Lustre: DEBUG MARKER: == sanity test 105h: Flock functional verify ============= 19:58:46 (1746143926) [ 2964.887460] Lustre: DEBUG MARKER: == sanity test 105i: Flock deadlock verify =============== 19:58:50 (1746143930) [ 2973.094682] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 19:58:58 (1746143938) [ 2976.116972] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 19:59:01 (1746143941) [ 2981.351276] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 19:59:07 (1746143947) [ 2985.120971] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 19:59:10 (1746143950) [ 3110.197409] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 20:01:16 (1746144076) [ 3111.330898] Lustre: *** cfs_fail_loc=147, val=0*** [ 3115.024927] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 20:01:20 (1746144080) [ 3118.303259] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 20:01:24 (1746144084) [ 3121.488552] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 20:01:27 (1746144087) [ 3122.214232] Lustre: *** cfs_fail_loc=217, val=0*** [ 3122.217044] Lustre: Skipped 31 previous similar messages [ 3125.139452] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 3125.879170] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 3126.567298] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 20:01:32 (1746144092) [ 3130.095436] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 20:01:35 (1746144095) [ 3133.375377] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 20:01:39 (1746144099) [ 3134.222398] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3135.248447] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3136.489968] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3140.456903] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3144.468393] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3147.982244] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 20:01:53 (1746144113) [ 3158.125873] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 20:02:03 (1746144123) [ 3159.153653] Lustre: *** cfs_fail_loc=220, val=0*** [ 3159.155297] Lustre: Skipped 3 previous similar messages [ 3163.710319] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 20:02:09 (1746144129) [ 3177.870439] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 20:02:23 (1746144143) [ 3181.030469] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 20:02:26 (1746144146) [ 3184.216466] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 20:02:30 (1746144150) [ 3190.355769] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 20:02:36 (1746144156) [ 3193.241997] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 20:02:39 (1746144159) [ 3196.273892] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 20:02:42 (1746144162) [ 3199.084237] Lustre: DEBUG MARKER: == sanity test 119e: Basic tests of dio read and write at various sizes ========================================================== 20:02:44 (1746144164) [ 3201.058234] Lustre: DEBUG MARKER: sanity test_119e: @@@@@@ FAIL: trivial unaligned dio failed [ 3205.252987] Lustre: DEBUG MARKER: == sanity test 119f: dio vs dio race ===================== 20:02:51 (1746144171) [ 3214.363185] Lustre: DEBUG MARKER: sanity test_119f: @@@@@@ FAIL: dio copy 1 w/bsize 1049600 failed: 1 [ 3218.367503] Lustre: DEBUG MARKER: == sanity test 119g: dio vs buffered I/O race ============ 20:03:04 (1746144184) [ 3230.359987] Lustre: DEBUG MARKER: sanity test_119g: @@@@@@ FAIL: dio copy 1 w/bsize 1049600 failed: 1 [ 3234.149223] Lustre: DEBUG MARKER: == sanity test 119h: basic tests of memory unaligned dio ========================================================== 20:03:19 (1746144199) [ 3235.583744] Lustre: DEBUG MARKER: sanity test_119h: @@@@@@ FAIL: multiop memory unaligned write failed, 1044480 [ 3238.368861] Lustre: DEBUG MARKER: SKIP: sanity test_119i skipping ALWAYS excluded test 119i [ 3239.119979] Lustre: DEBUG MARKER: == sanity test 119j: basic tests of hybrid IO switching == 20:03:25 (1746144205) [ 3239.786843] Lustre: DEBUG MARKER: SKIP: sanity test_119j needs kernel > 4.5.0 for ki_flags support [ 3240.565999] Lustre: DEBUG MARKER: == sanity test 119m: Test DIO readv/writev: exercise iter duplication ========================================================== 20:03:26 (1746144206) [ 3243.675592] Lustre: DEBUG MARKER: == sanity test 119n: Test Unaligned DIO readv() and writev() with unpatched ZFS ========================================================== 20:03:29 (1746144209) [ 3244.339368] Lustre: DEBUG MARKER: SKIP: sanity test_119n need ZFS server without unaligned_dio support [ 3245.264868] Lustre: DEBUG MARKER: == sanity test 119o: Test Unaligned DIO readv() and writev() with unpatched servers ========================================================== 20:03:31 (1746144211) [ 3246.110481] Lustre: DEBUG MARKER: SKIP: sanity test_119o need ldiskfs without unaligned_dio support. [ 3247.158817] Lustre: DEBUG MARKER: == sanity test 119p: Test Unaligned DIO readv() and writev() with patched servers ========================================================== 20:03:32 (1746144212) [ 3248.091331] Lustre: DEBUG MARKER: sanity test_119p: @@@@@@ FAIL: DIO unaligned writev test failed [ 3251.615455] Lustre: DEBUG MARKER: == sanity test 119q: Test patchded Unaligned DIO readv() and writev() ========================================================== 20:03:37 (1746144217) [ 3252.594294] Lustre: DEBUG MARKER: sanity test_119q: @@@@@@ FAIL: DIO unaligned writev test failed: 69631 [ 3255.104832] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 20:03:40 (1746144220) [ 3259.590877] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 20:03:45 (1746144225) [ 3264.044659] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 20:03:49 (1746144229) [ 3269.189551] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 20:03:55 (1746144235) [ 3274.407683] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 20:04:00 (1746144240) [ 3286.981525] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 20:04:12 (1746144252) [ 3299.424418] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 20:04:24 (1746144264) [ 3529.105538] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 20:08:15 (1746144495) [ 3531.896941] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 20:08:17 (1746144497) [ 3536.253052] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 2 sec [ 3537.460233] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 3555.298694] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 11 sec [ 3557.769761] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 1 sec [ 3623.180424] Lustre: DEBUG MARKER: ls -l 4000 files without statahead: 42 sec [ 3631.372860] Lustre: DEBUG MARKER: ls -l 4000 files with statahead: 7 sec [ 3631.949586] Lustre: DEBUG MARKER: ls -l done [ 3663.347548] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity: 31 seconds [ 3667.091373] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 20:10:32 (1746144632) [ 3667.813642] Lustre: DEBUG MARKER: SKIP: sanity test_123ab Test must be statx() syscall supported [ 3668.800319] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 20:10:34 (1746144634) [ 3669.640523] Lustre: DEBUG MARKER: SKIP: sanity test_123ac Test must be statx() syscall supported [ 3670.497764] Lustre: DEBUG MARKER: == sanity test 123ad: Verify batching statahead works correctly ========================================================== 20:10:36 (1746144636) [ 3675.188712] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 2 sec [ 3676.255333] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 3697.078713] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 12 sec [ 3702.015061] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 4 sec [ 3770.452987] Lustre: DEBUG MARKER: ls -l 4000 files without statahead: 46 sec [ 3785.337552] Lustre: DEBUG MARKER: ls -l 4000 files with statahead: 13 sec [ 3785.976765] Lustre: DEBUG MARKER: ls -l done [ 3819.771816] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 33 seconds [ 4091.082798] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 4092.190002] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 4113.268659] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 12 sec [ 4116.869510] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 2 sec [ 4189.302718] Lustre: DEBUG MARKER: ls -l 4000 files without statahead: 48 sec [ 4198.641593] Lustre: DEBUG MARKER: ls -l 4000 files with statahead: 7 sec [ 4199.277116] Lustre: DEBUG MARKER: ls -l done [ 4233.875847] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 34 seconds [ 4457.425575] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 20:23:43 (1746145423) [ 4469.831414] Lustre: DEBUG MARKER: ls done [ 4483.435629] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 20:24:09 (1746145449) [ 4488.571395] Lustre: DEBUG MARKER: == sanity test 123d: Statahead on striped directories works correctly ========================================================== 20:24:14 (1746145454) [ 4501.047347] Lustre: DEBUG MARKER: == sanity test 123e: statahead with large wide striping == 20:24:26 (1746145466) [ 4584.089658] Lustre: 4864:0:(mdt_handler.c:4672:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 4643.705577] hrtimer: interrupt took 2897071 ns [ 4656.907614] Lustre: DEBUG MARKER: == sanity test 123f: Retry mechanism with large wide striping files ========================================================== 20:27:02 (1746145622) [ 4717.369976] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4717.379112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4847.342927] Lustre: lustre-MDT0000: Client 4d1ff808-81ea-462d-8672-4c7065432c42 (at 192.168.203.1@tcp) reconnecting [ 4847.352479] Lustre: Skipped 1 previous similar message [ 4853.786805] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4854.474852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4871.788717] Lustre: 4864:0:(mdt_handler.c:4672:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 4942.753957] Lustre: 7228:0:(mdt_handler.c:4672:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 4942.758676] Lustre: 7228:0:(mdt_handler.c:4672:mdt_unpack_req_pack_rep()) Skipped 1 previous similar message [ 5035.594803] Lustre: 7223:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 21312 > trans_max 3200 [ 5035.598557] Lustre: 7223:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 5035.602564] Lustre: 7223:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1399/1399/0, xattr_set: 2098/19692/0 [ 5035.607116] Lustre: 7223:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 5035.613210] Lustre: 7223:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 5035.618153] Lustre: 7223:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 5035.622295] CPU: 2 PID: 7223 Comm: mdt00_002 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 5035.626078] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 5035.629341] Call Trace: [ 5035.631146] [] dump_stack+0x19/0x1b [ 5035.633492] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 5035.636561] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 5035.639669] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 5035.644421] [] lod_trans_start+0x7f/0x2e0 [lod] [ 5035.648315] [] mdd_trans_start+0x14/0x20 [mdd] [ 5035.650458] [] mdd_unlink+0x617/0xea0 [mdd] [ 5035.652883] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 5035.655925] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 5035.660434] [] mdt_reint_rec+0x87/0x240 [mdt] [ 5035.663576] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 5035.665889] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 5035.668594] [] mdt_reint+0x67/0x150 [mdt] [ 5035.671067] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 5035.674489] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 5035.677688] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 5035.682281] [] ? put_prev_entity+0x31/0x400 [ 5035.685769] [] ? do_raw_spin_unlock+0x49/0x90 [ 5035.690475] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 5035.696266] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 5035.704783] [] kthread+0xe4/0xf0 [ 5035.707634] [] ? kthread_create_on_node+0x140/0x140 [ 5035.710667] [] ret_from_fork_nospec_begin+0x7/0x21 [ 5035.712961] [] ? kthread_create_on_node+0x140/0x140 [ 5079.782467] Lustre: DEBUG MARKER: == sanity test 123g: Test for stat-ahead advise ========== 20:34:05 (1746146045) [ 5105.942011] Lustre: DEBUG MARKER: == sanity test 123h: Verify statahead work with the fname pattern via du ========================================================== 20:34:31 (1746146071) [ 5750.436322] Lustre: DEBUG MARKER: == sanity test 123i: Verify statahead work with the fname indexing pattern ========================================================== 20:45:16 (1746146716) [ 5828.511909] Lustre: DEBUG MARKER: == sanity test 123j: -ENOENT error from batched statahead be handled correctly ========================================================== 20:46:34 (1746146794) [ 5831.454445] Lustre: DEBUG MARKER: == sanity test 123k: Verify statahead work with mdtest shared stat() mode ========================================================== 20:46:37 (1746146797) [ 5831.997347] Lustre: DEBUG MARKER: SKIP: sanity test_123k mdtest not found [ 5832.635179] Lustre: DEBUG MARKER: == sanity test 123l: Avoid panic when revalidate a local cached entry ========================================================== 20:46:38 (1746146798) [ 5903.963398] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 20:47:49 (1746146869) [ 5904.563392] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 5922.647463] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff88012b55e800 [ 5923.303500] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff88012b55e800 [ 5923.986563] Lustre: DEBUG MARKER: LRU=1015 [ 5924.722872] Lustre: DEBUG MARKER: LIMIT=46624 [ 5925.436281] Lustre: DEBUG MARKER: LVF=5512100 [ 5926.166376] Lustre: DEBUG MARKER: OLD_LVF=100 [ 5926.791441] Lustre: DEBUG MARKER: Sleep 50 sec [ 5977.386777] Lustre: DEBUG MARKER: Dropped 430 locks in 50s [ 5977.903114] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 5986.634431] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 20:49:12 (1746146952) [ 6027.190589] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 6099.813517] Lustre: DEBUG MARKER: ls -la time: 72 seconds [ 6100.566021] Lustre: DEBUG MARKER: lru_size = 400 [ 6181.265379] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 6210.239186] Lustre: DEBUG MARKER: ls -la time: 27 seconds [ 6211.034296] Lustre: DEBUG MARKER: lru_size = 4006 [ 6211.816712] Lustre: DEBUG MARKER: ls -la is 62% faster with lru resize enabled [ 6248.257069] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 20:53:34 (1746147214) [ 6275.114835] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize disabled ========================================================== 20:54:00 (1746147240) [ 6301.754908] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 20:54:27 (1746147267) [ 6305.769441] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 20:54:31 (1746147271) [ 6309.890422] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 20:54:35 (1746147275) [ 6314.669882] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 20:54:40 (1746147280) [ 6318.220231] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [ 6334.511299] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 20:55:00 (1746147300) [ 6337.628513] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 20:55:03 (1746147303) [ 6346.673946] Lustre: 12024:0:(osd_handler.c:600:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513297, FID: [0x20000040c:0x82eb:0x0]) is approaching max size limit [ 6347.956220] Lustre: 7222:0:(osd_handler.c:600:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513297, FID: [0x20000040c:0x82eb:0x0]) is approaching max size limit [ 6354.316799] Lustre: 12024:0:(osd_handler.c:596:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513297, FID: [0x20000040c:0x82eb:0x0]) has reached max size limit [ 6367.502810] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 20:55:33 (1746147333) [ 6370.345702] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 20:55:36 (1746147336) [ 6373.042570] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 20:55:39 (1746147339) [ 6375.799506] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 20:55:41 (1746147341) [ 6376.380522] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [ 6376.988396] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 20:55:43 (1746147343) [ 6403.193534] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 20:56:09 (1746147369) [ 6405.754663] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 20:56:11 (1746147371) [ 6418.267474] Lustre: 10082:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 6372 > trans_max 3200 [ 6418.271019] Lustre: 10082:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 6418.274257] Lustre: 10082:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 403/403/0, xattr_set: 604/5748/0 [ 6418.279462] Lustre: 10082:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 6418.283228] Lustre: 10082:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 6418.288137] Lustre: 10082:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 6418.293592] CPU: 3 PID: 10082 Comm: mdt00_003 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 6418.297968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 6418.302154] Call Trace: [ 6418.303202] [] dump_stack+0x19/0x1b [ 6418.305385] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 6418.308606] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 6418.311426] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 6418.315306] [] lod_trans_start+0x7f/0x2e0 [lod] [ 6418.317662] [] mdd_trans_start+0x14/0x20 [mdd] [ 6418.320246] [] mdd_unlink+0x617/0xea0 [mdd] [ 6418.322115] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 6418.324678] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 6418.326986] [] mdt_reint_rec+0x87/0x240 [mdt] [ 6418.329887] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 6418.333778] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 6418.337875] [] mdt_reint+0x67/0x150 [mdt] [ 6418.340310] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 6418.342640] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 6418.348057] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 6418.351455] [] ? do_raw_spin_unlock+0x49/0x90 [ 6418.353479] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 6418.355520] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 6418.357565] [] kthread+0xe4/0xf0 [ 6418.358914] [] ? kthread_create_on_node+0x140/0x140 [ 6418.361025] [] ret_from_fork_nospec_begin+0x7/0x21 [ 6418.363172] [] ? kthread_create_on_node+0x140/0x140 [ 6420.383988] Lustre: DEBUG MARKER: == sanity test 130h: FIEMAP deadlock ===================== 20:56:26 (1746147386) [ 6428.829483] Lustre: DEBUG MARKER: == sanity test 130i: FIEMAP (DoM file) =================== 20:56:34 (1746147394) [ 6445.785018] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 20:56:51 (1746147411) [ 6447.889131] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 20:56:53 (1746147413) [ 6450.128124] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 20:56:56 (1746147416) [ 6452.039150] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 20:56:58 (1746147418) [ 6454.386695] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 20:57:00 (1746147420) [ 6456.626463] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 20:57:02 (1746147422) [ 6463.611326] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 20:57:09 (1746147429) [ 6467.875259] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 20:57:13 (1746147433) [ 6482.744065] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 20:57:28 (1746147448) [ 6498.957361] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 20:57:44 (1746147464) [ 6504.377377] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 20:57:50 (1746147470) [ 6515.558781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6515.560532] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6515.560543] Lustre: Skipped 1 previous similar message [ 6515.560924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6517.988626] LustreError: 26959:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6517.992483] LustreError: 26959:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6518.081232] Lustre: server umount lustre-MDT0000 complete [ 6520.568372] LustreError: 19891:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.576757] LustreError: 19891:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 6521.346860] LustreError: 7551:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746147487 with bad export cookie 12641298230503428739 [ 6521.349572] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6521.367380] LustreError: 7551:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6521.441179] LustreError: 27561:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6521.443833] LustreError: 27561:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6521.555611] Lustre: server umount lustre-MDT0001 complete [ 6534.625208] LustreError: 28154:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6534.627227] LustreError: 28154:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6534.710594] Lustre: server umount lustre-OST0000 complete [ 6548.235650] LustreError: 28757:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6548.238438] LustreError: 28757:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6548.408710] Lustre: server umount lustre-OST0001 complete [ 6550.893546] device-mapper: core: cleaned up [ 6554.214969] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 6555.118413] Key type lgssc unregistered [ 6555.263065] LNet: 29620:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6556.269926] LNet: Removed LNI 192.168.203.101@tcp [ 6561.524121] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6561.730506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6561.756456] alg: No test for adler32 (adler32-zlib) [ 6562.694613] Lustre: Lustre: Build Version: 2.16.54_75_gccae094 [ 6562.779717] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 6562.782689] LNet: Accept secure, port 988 [ 6564.336550] Key type lgssc registered [ 6564.706687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6569.395419] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6570.695106] device-mapper: uevent: version 1.0.3 [ 6570.697003] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 6573.052620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6573.065554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6574.260737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6575.261140] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [ 6579.176523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6579.232961] Lustre: 426:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6579.401314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6579.456549] mount.lustre (426) used greatest stack depth: 9584 bytes left [ 6580.818758] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [ 6582.284263] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 6586.417474] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6586.426614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6586.547547] Lustre: 1589:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6586.660350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6588.767207] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [ 6589.662077] LustreError: 1595:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6589.669785] LustreError: 1595:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6589.674338] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:11329 to 0x280000400:11361) [ 6593.743584] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6593.751889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6593.822497] Lustre: 2785:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6593.924923] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6596.128016] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [ 6597.077697] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:11494 to 0x2c0000400:11521) [ 6599.078587] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23410 to 0x2c0000403:23553) [ 6599.087664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:22241 to 0x280000403:22401) [ 6604.768819] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6606.230172] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 6614.541408] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 20:59:40 (1746147580) [ 6639.220157] LNet: 5523:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 6639.264041] Lustre: DEBUG MARKER:  [ 6639.266728] Lustre: DEBUG MARKER:  [ 6639.863675] LNet: 5624:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 6639.869338] LNet: 5624:0:(debug.c:388:cfs_str2mask()) Skipped 1 previous similar message [ 6666.306862] LNet: 7137:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 6666.312037] LNet: 7137:0:(debug.c:388:cfs_str2mask()) Skipped 1 previous similar message [ 6666.339704] Lustre: DEBUG MARKER:  [ 6666.341156] Lustre: DEBUG MARKER:  [ 6669.223106] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.223959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6669.232630] Lustre: Skipped 3 previous similar messages [ 6674.231284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6674.233883] Lustre: Skipped 6 previous similar messages [ 6674.894900] LustreError: 7413:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6674.948909] Lustre: server umount lustre-MDT0000 complete [ 6678.413217] LustreError: 925:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746147644 with bad export cookie 14181480685189329771 [ 6678.420005] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6678.489449] LustreError: 8013:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6678.492456] LustreError: 8013:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6678.584413] Lustre: server umount lustre-MDT0001 complete [ 6691.740763] LustreError: 8607:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6691.742919] LustreError: 8607:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6691.766279] Lustre: server umount lustre-OST0000 complete [ 6704.718570] LustreError: 9211:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6704.720676] LustreError: 9211:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6704.776953] Lustre: server umount lustre-OST0001 complete [ 6706.913697] device-mapper: core: cleaned up [ 6710.858811] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 6711.694154] Key type lgssc unregistered [ 6711.793793] LNet: 10139:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6712.804914] LNet: Removed LNI 192.168.203.101@tcp [ 6717.636623] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6717.767107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6717.802910] alg: No test for adler32 (adler32-zlib) [ 6718.632392] Lustre: Lustre: Build Version: 2.16.54_75_gccae094 [ 6718.691303] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 6718.693583] LNet: Accept secure, port 988 [ 6720.231248] Key type lgssc registered [ 6720.469844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6724.648783] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 6725.955391] device-mapper: uevent: version 1.0.3 [ 6725.957672] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 6728.217673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6728.227948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.505601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6730.907953] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [ 6735.543591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6735.622645] Lustre: 13416:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6735.827690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6737.120471] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [ 6738.447270] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 6742.665658] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6742.675727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6742.811706] Lustre: 14542:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6742.909012] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6742.943607] mount.lustre (14542) used greatest stack depth: 9504 bytes left [ 6745.147487] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [ 6745.906326] LustreError: 14547:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6745.920855] LustreError: 14547:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6745.925399] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:11329 to 0x280000400:11393) [ 6749.599404] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6749.607992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6749.666336] Lustre: 15732:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6749.752270] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6751.500619] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [ 6753.114961] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:11494 to 0x2c0000400:11553) [ 6754.107577] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23410 to 0x2c0000403:23585) [ 6754.114244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:22241 to 0x280000403:22433) [ 6759.981561] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6761.391162] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 6769.822936] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 21:02:15 (1746147735) [ 7201.321871] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 21:09:27 (1746148167) [ 7208.576297] Lustre: *** cfs_fail_loc=327, val=500*** [ 7230.371064] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 21:09:56 (1746148196) [ 7233.655936] Lustre: *** cfs_fail_loc=328, val=500*** [ 7233.662163] Lustre: Skipped 1167 previous similar messages [ 7234.658146] Lustre: *** cfs_fail_loc=328, val=500*** [ 7234.659710] Lustre: Skipped 141 previous similar messages [ 7235.297418] Lustre: 12342:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8800a8fc9880 x1830968550871808/t0(0) o101->0785784f-96ff-4744-84ff-e1cfe627e335@192.168.203.1@tcp:67/0 lens 648/0 e 0 to 0 dl 1746148212 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7236.660052] Lustre: *** cfs_fail_loc=328, val=500*** [ 7236.661811] Lustre: Skipped 150 previous similar messages [ 7237.041820] Lustre: 22421:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff88012c3d2300 x1830968550936576/t0(0) o101->0785784f-96ff-4744-84ff-e1cfe627e335@192.168.203.1@tcp:69/0 lens 648/0 e 0 to 0 dl 1746148214 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7238.065951] Lustre: 22421:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff88009fceca80 x1830968550937728/t0(0) o101->0785784f-96ff-4744-84ff-e1cfe627e335@192.168.203.1@tcp:70/0 lens 648/0 e 0 to 0 dl 1746148215 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7240.118007] Lustre: 16279:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff88009fcedf80 x1830968550939264/t0(0) o101->0785784f-96ff-4744-84ff-e1cfe627e335@192.168.203.1@tcp:72/0 lens 648/0 e 0 to 0 dl 1746148217 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7240.129889] Lustre: 16279:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) Skipped 1 previous similar message [ 7241.156316] Lustre: *** cfs_fail_loc=328, val=500*** [ 7241.160881] Lustre: Skipped 87 previous similar messages [ 7244.476629] Lustre: 16279:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff88008c212680 x1830968550972928/t0(0) o101->0785784f-96ff-4744-84ff-e1cfe627e335@192.168.203.1@tcp:76/0 lens 648/0 e 0 to 0 dl 1746148221 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7244.490981] Lustre: 16279:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) Skipped 3 previous similar messages [ 7249.708752] Lustre: *** cfs_fail_loc=328, val=500*** [ 7249.722478] Lustre: Skipped 93 previous similar messages [ 7253.585164] LustreError: 24565:0:(ldlm_lockd.c:3258:lock_reclaim_threshold_mb_store()) Failed to set lock_reclaim_threshold_mb, rc = -22. [ 7261.654166] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [ 7262.570135] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [ 7263.431479] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 21:10:29 (1746148229) [ 7287.931556] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 21:10:53 (1746148253) [ 7308.285768] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 21:11:13 (1746148273) [ 7331.493948] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 21:11:36 (1746148296) [ 7353.348582] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 21:11:59 (1746148319) [ 7363.135423] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 21:12:08 (1746148328) [ 7372.796055] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 21:12:18 (1746148338) [ 7385.603041] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 21:12:31 (1746148351) [ 7395.484695] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 21:12:41 (1746148361) [ 7410.241841] Lustre: DEBUG MARKER: == sanity test 150h: Verify extend fallocate updates the file size ========================================================== 21:12:56 (1746148376) [ 7411.428075] Lustre: DEBUG MARKER: SKIP: sanity test_150h Test must be statx() syscall supported [ 7412.324820] Lustre: DEBUG MARKER: == sanity test 150ia: Verify fallocate zero-range ZERO functionality ========================================================== 21:12:58 (1746148378) [ 7422.639500] Lustre: DEBUG MARKER: == sanity test 150ib: Verify fallocate zero-range PREALLOC functionality ========================================================== 21:13:08 (1746148388) [ 7431.843437] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 21:13:17 (1746148397) [ 7438.805052] bash (2181): drop_caches: 1 [ 7444.398104] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 21:13:30 (1746148410) [ 7447.773953] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 21:13:33 (1746148413) [ 7451.007183] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 21:13:36 (1746148416) [ 7454.845460] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 21:13:40 (1746148420) [ 7459.322307] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 21:13:44 (1746148424) [ 7460.308724] LustreError: 16279:0:(fld_handler.c:244:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 7465.762604] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 21:13:51 (1746148431) [ 7466.798656] LustreError: 16279:0:(fld_handler.c:244:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 7466.803450] LustreError: 16279:0:(fld_handler.c:244:fld_server_lookup()) Skipped 3 previous similar messages [ 7471.082224] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 21:13:56 (1746148436) [ 7474.811296] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 21:14:00 (1746148440) [ 7479.817438] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 21:14:05 (1746148445) [ 7483.135956] Lustre: DEBUG MARKER: == sanity test 154ea: .lustre is not returned by readdir (2) ========================================================== 21:14:08 (1746148448) [ 7515.222820] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 21:14:41 (1746148481) [ 7519.674413] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 21:14:45 (1746148485) [ 8039.852195] Lustre: DEBUG MARKER: == sanity test 154h: Verify interactive path2fid ========= 21:23:25 (1746149005) [ 8042.725043] Lustre: DEBUG MARKER: == sanity test 154i: fid2path for path longer than PATH_MAX ========================================================== 21:23:28 (1746149008) [ 8064.252007] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 21:23:49 (1746149029) [ 8071.521413] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 21:23:57 (1746149037) [ 8078.024583] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 21:24:03 (1746149043) [ 8085.039715] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 21:24:10 (1746149050) [ 8092.654433] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 21:24:18 (1746149058) [ 8120.576520] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 21:24:46 (1746149086) [ 8152.328184] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 21:25:17 (1746149117) [ 8183.970032] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 21:25:49 (1746149149) [ 8215.245507] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 21:26:20 (1746149180) [ 8220.962922] Lustre: DEBUG MARKER: Turn on read and write cache [ 8222.943374] Lustre: DEBUG MARKER: Write data and read it back. [ 8223.785836] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 8226.168337] Lustre: DEBUG MARKER: cache hits: before: 65582, after: 65585 [ 8226.980565] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8228.524855] Lustre: DEBUG MARKER: cache hits:: before: 65585, after: 65588 [ 8229.379726] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [ 8231.647829] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8233.828258] Lustre: DEBUG MARKER: cache hits:: before: 65588, after: 65591 [ 8234.732319] Lustre: DEBUG MARKER: Write data and read it back. [ 8235.531891] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 8237.851985] Lustre: DEBUG MARKER: cache hits:: before: 65591, after: 65594 [ 8238.833676] Lustre: DEBUG MARKER: Turn off read and write cache [ 8241.132570] Lustre: DEBUG MARKER: Write data and read it back [ 8242.167013] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 8244.768191] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65594 [ 8245.861719] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [ 8247.815989] Lustre: DEBUG MARKER: Write data and read it back [ 8248.594473] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 8250.553198] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65594 [ 8251.347873] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8253.312489] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65597 [ 8257.784904] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 21:27:03 (1746149223) [ 8259.410028] Lustre: lustre-MDD0000: changelog on [ 8260.942723] Lustre: lustre-MDD0001: changelog on [ 8270.774145] Lustre: Failing over lustre-MDT0000 [ 8270.912586] LustreError: 18537:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8271.019956] Lustre: server umount lustre-MDT0000 complete [ 8273.447823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8273.456329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8273.482542] LustreError: 12347:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8274.114347] LustreError: 13433:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8274.854069] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8274.859845] Lustre: Skipped 2 previous similar messages [ 8275.713988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8275.812584] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8276.008852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8276.047752] Lustre: lustre-MDD0000: changelog on [ 8276.070956] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8277.933373] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [ 8279.112181] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8281.022738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.203.101@tcp (at 0@lo) [ 8281.222476] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 8281.266597] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:24420 to 0x2c0000403:24449) [ 8281.270434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:23271 to 0x280000403:23297) [ 8281.673368] Lustre: lustre-MDD0000: changelog off [ 8282.693008] Lustre: lustre-MDD0001: changelog off [ 8290.296959] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 21:27:36 (1746149256) [ 8291.833388] Lustre: lustre-MDD0000: changelog on [ 8297.103644] Lustre: lustre-MDD0001: changelog off [ 8299.390042] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 21:27:45 (1746149265) [ 8300.685889] Lustre: lustre-MDD0000: changelog on [ 8300.687871] Lustre: Skipped 1 previous similar message [ 8306.760145] Lustre: lustre-MDD0001: changelog off [ 8306.761683] Lustre: Skipped 1 previous similar message [ 8309.374216] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 21:27:55 (1746149275) [ 8310.713753] Lustre: lustre-MDD0000: changelog on [ 8310.716283] Lustre: Skipped 1 previous similar message [ 8316.072850] Lustre: lustre-MDD0001: changelog off [ 8316.074520] Lustre: Skipped 1 previous similar message [ 8318.933837] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 21:28:04 (1746149284) [ 8326.148888] Lustre: lustre-MDD0001: changelog off [ 8326.150453] Lustre: Skipped 1 previous similar message [ 8328.924947] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 21:28:14 (1746149294) [ 8330.741645] Lustre: lustre-MDD0000: changelog on [ 8330.743640] Lustre: Skipped 3 previous similar messages [ 8335.878108] Lustre: DEBUG MARKER: 1746149301: creating first dirs [ 8351.908250] Lustre: *** cfs_fail_loc=1313, val=3*** [ 8351.911390] Lustre: 22421:0:(mdd_dir.c:911:mdd_changelog_emrg_cleanup()) lustre-MDD0000: changelog has only 3 free catalog entries [ 8351.919903] Lustre: 22421:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 8351.925576] Lustre: 27032:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl7 idle for 18s with 4 unprocessed records [ 8362.793747] Lustre: lustre-MDD0001: changelog off [ 8362.795430] Lustre: Skipped 1 previous similar message [ 8365.507997] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 21:28:51 (1746149331) [ 8366.915628] Lustre: lustre-MDD0000: changelog on [ 8366.917516] Lustre: Skipped 1 previous similar message [ 8378.422920] Lustre: 13433:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 8378.429155] Lustre: 13433:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 8378.441883] Lustre: 30332:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl9 idle for 9s with 4 unprocessed records [ 8378.450697] Lustre: 30332:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 8390.652801] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 21:29:16 (1746149356) [ 8413.192278] Lustre: *** cfs_fail_loc=1316, val=0*** [ 8413.196296] Lustre: 12343:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0001: simulate starting changelog garbage collection [ 8413.203399] Lustre: 12343:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 8413.211704] Lustre: 1199:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0001: force deregister of changelog user cl11 idle for 19s with 3 unprocessed records [ 8413.221943] Lustre: 1199:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 2 previous similar messages [ 8414.259083] Lustre: Failing over lustre-MDT0000 [ 8414.374913] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 8414.377568] Lustre: Failing over lustre-MDT0001 [ 8416.249762] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8416.251317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8416.262941] Lustre: Skipped 1 previous similar message [ 8418.420600] LustreError: 1473:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8418.424625] LustreError: 1473:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8418.472717] Lustre: server umount lustre-MDT0000 complete [ 8418.552215] Lustre: server umount lustre-MDT0001 complete [ 8422.235288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8422.308687] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8422.460973] LustreError: 14557:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8422.471920] LustreError: 14557:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 8422.527401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8422.560073] Lustre: 2729:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 29 to index 30, being cleared now [ 8422.576071] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8423.951334] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [ 8426.527443] LustreError: 14557:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8426.542273] LustreError: 14557:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 8427.576910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8427.690847] Lustre: 3616:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8427.798031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 8427.902251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8427.930016] Lustre: 3616:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0001 : orphan changelog records found, starting from index 17 to index 18, being cleared now [ 8427.953854] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8429.358684] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [ 8429.529196] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [ 8432.904696] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8434.240311] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 192.168.203.101@tcp (at 0@lo) [ 8434.240741] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8434.256736] Lustre: Skipped 3 previous similar messages [ 8434.258419] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:11400 to 0x280000400:11425) [ 8434.259891] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:11560 to 0x2c0000400:11585) [ 8435.811226] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8435.811736] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 192.168.203.101@tcp (at 0@lo) [ 8435.847835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:23271 to 0x280000403:23329) [ 8435.847851] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:24451 to 0x2c0000403:24481) [ 8445.447276] Lustre: lustre-MDD0001: changelog off [ 8445.449448] Lustre: Skipped 3 previous similar messages [ 8448.571631] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 21:30:14 (1746149414) [ 8450.080804] Lustre: lustre-MDD0000: changelog on [ 8450.083798] Lustre: Skipped 5 previous similar messages [ 8454.524122] LustreError: 6179:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_race id 1315 sleeping [ 8456.773561] LustreError: 6349:0:(mdd_device.c:1746:mdd_changelog_user_register()) cfs_fail_race id 1315 waking [ 8456.776874] LustreError: 6179:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_fail_race id 1315 awake: rc=2751 [ 8457.650320] LustreError: 6602:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_fail_race id 1315 waking [ 8470.028264] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 21:30:35 (1746149435) [ 8483.752344] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 21:30:49 (1746149449) [ 8488.824597] LustreError: 4232:0:(mdd_dir.c:966:mdd_changelog_store()) cfs_fail_timeout id 15d sleeping for 3000ms [ 8491.837890] LustreError: 4232:0:(mdd_dir.c:966:mdd_changelog_store()) cfs_fail_timeout id 15d awake [ 8502.464603] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 21:31:08 (1746149468) [ 8511.725534] Lustre: lustre-MDD0001: changelog off [ 8511.727453] Lustre: Skipped 9 previous similar messages [ 8514.293388] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 21:31:20 (1746149480) [ 8522.657699] LustreError: 4238:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_race id 15f sleeping [ 8524.669330] LustreError: 2740:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_fail_race id 15f waking [ 8524.673050] LustreError: 2740:0:(mdd_device.c:394:llog_changelog_cancel_cb()) Skipped 1 previous similar message [ 8524.678797] LustreError: 4238:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_fail_race id 15f awake: rc=2982 [ 8533.085756] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 21:31:38 (1746149498) [ 9700.213830] LustreError: 14425:0:(mdd_device.c:408:llog_changelog_cancel_cb()) cfs_race id 16c sleeping [ 9702.230574] LustreError: 2740:0:(llog_osd.c:1136:llog_osd_next_block()) cfs_fail_race id 16c waking [ 9702.234878] LustreError: 14425:0:(mdd_device.c:408:llog_changelog_cancel_cb()) cfs_fail_race id 16c awake: rc=2987 [ 9708.829414] Lustre: lustre-MDD0001: changelog off [ 9708.831010] Lustre: Skipped 3 previous similar messages [ 9711.101144] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 21:51:17 (1746150677) [ 9712.468163] Lustre: lustre-MDD0000: changelog on [ 9712.469467] Lustre: Skipped 13 previous similar messages [ 9714.251556] LustreError: 17520:0:(mdd_device.c:1700:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [ 9714.637287] Lustre: 17585:0:(mdd_device.c:1717:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [ 9715.003213] LustreError: 17649:0:(mdd_device.c:1709:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [ 9727.651563] Lustre: lustre-MDD0000: changelog off [ 9727.654689] Lustre: Skipped 1 previous similar message [ 9731.139145] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 21:51:36 (1746150696) [ 9732.576515] Lustre: lustre-MDD0000: changelog on [ 9732.578452] Lustre: Skipped 1 previous similar message [ 9736.105340] Lustre: Failing over lustre-MDT0000 [ 9736.288583] LustreError: 20102:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9736.299153] LustreError: 20102:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 9736.395188] Lustre: server umount lustre-MDT0000 complete [ 9737.946884] LustreError: 2740:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9738.134780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9738.149966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9739.190563] LustreError: 4233:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9739.191130] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9739.202149] LustreError: 4233:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 9741.772716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9741.845310] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9742.027991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9742.104700] Lustre: 20765:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 90153 to index 18446744073709551615, being cleared now [ 9742.135581] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9742.949317] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9743.807926] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [ 9747.034786] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.203.101@tcp (at 0@lo) [ 9747.043195] Lustre: Skipped 4 previous similar messages [ 9747.062994] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9747.131224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:23331 to 0x280000403:23361) [ 9747.131313] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:24483 to 0x2c0000403:24513) [ 9753.165651] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 21:51:59 (1746150719) [ 9757.986621] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records * time ========================================================== 21:52:03 (1746150723) [ 9766.596228] Lustre: 4238:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 9766.601147] Lustre: 4238:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 9766.610657] Lustre: 23342:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl2 idle for 864007s with 500000004 unprocessed records [ 9766.624945] Lustre: lustre-MDD0000: changelog off [ 9766.628215] Lustre: Skipped 3 previous similar messages [ 9775.908906] Lustre: DEBUG MARKER: == sanity test 160t: changelog garbage collect on lack of space ========================================================== 21:52:21 (1746150741) [ 9777.469916] Lustre: lustre-MDD0000: changelog on [ 9777.471883] Lustre: Skipped 4 previous similar messages [ 9809.362004] Lustre: *** cfs_fail_loc=18c, val=1211180*** [ 9809.364059] Lustre: Skipped 1 previous similar message [ 9809.365953] Lustre: 4232:0:(mdd_dir.c:881:mdd_changelog_is_space_safe()) lustre-MDD0000: changelog uses 21MB with 1MB space limit [ 9809.371264] Lustre: 4232:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 9809.376160] Lustre: 4232:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 9809.382694] Lustre: 26011:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl3-user1 idle for 32s with 7506 unprocessed records [ 9809.399485] Lustre: 26011:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 9809.881611] Lustre: *** cfs_fail_loc=18c, val=1211180*** [ 9832.220007] Lustre: DEBUG MARKER: == sanity test 160u: changelog rename record type name and sname strings are correct ========================================================== 21:53:18 (1746150798) [ 9841.182915] Lustre: lustre-MDD0001: changelog off [ 9841.184882] Lustre: Skipped 3 previous similar messages [ 9844.061642] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 21:53:29 (1746150809) [ 9866.678722] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 21:53:52 (1746150832) [ 9890.731316] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 21:54:16 (1746150856) [ 9892.175661] Lustre: lustre-MDD0000: changelog on [ 9892.177781] Lustre: Skipped 3 previous similar messages [ 9903.044083] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 21:54:28 (1746150868) [ 9915.215782] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 21:54:40 (1746150880) [ 9920.438003] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 21:54:46 (1746150886) [ 9925.041919] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 21:54:50 (1746150890) [ 9995.787346] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 21:56:01 (1746150961) [10002.545703] Lustre: Failing over lustre-OST0000 [10002.606243] LustreError: 742:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10002.613206] LustreError: 742:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10002.662598] Lustre: server umount lustre-OST0000 complete [10002.666027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10002.674395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10002.685385] Lustre: Skipped 2 previous similar messages [10002.687218] LustreError: 23664:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10007.147401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10007.159338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10007.167812] LustreError: 23664:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10007.180413] LustreError: 23664:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [10012.456532] LustreError: 23662:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10012.466322] LustreError: 23662:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [10012.777454] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10012.785284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10012.869468] Lustre: 1908:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [10013.013025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10013.022059] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10013.373235] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10014.299071] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10014.302198] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.101@tcp (at 0@lo) [10014.302200] Lustre: Skipped 3 previous similar messages [10015.414966] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [10017.487754] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 21:56:23 (1746150983) [10044.842046] Lustre: Failing over lustre-OST0000 [10044.882774] LustreError: 3302:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10044.892891] LustreError: 3302:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10044.925294] Lustre: server umount lustre-OST0000 complete [10048.073232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10048.087903] LustreError: 23664:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10048.093584] LustreError: 23664:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [10049.860951] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10049.875129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10049.971433] Lustre: 3911:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [10050.145019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10050.157881] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10051.205234] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10052.112369] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10052.115028] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.101@tcp (at 0@lo) [10052.115845] Lustre: Skipped 1 previous similar message [10053.013314] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [10055.186856] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 21:57:00 (1746151020) [10073.655155] Lustre: Failing over lustre-OST0000 [10073.693677] LustreError: 5302:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10073.698753] LustreError: 5302:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10073.729393] Lustre: server umount lustre-OST0000 complete [10075.176018] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10075.190019] Lustre: Skipped 1 previous similar message [10075.196870] LustreError: 5362:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10075.208644] LustreError: 5362:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [10077.779864] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10077.792131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10077.850337] Lustre: 5905:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [10077.950959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10077.962494] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10078.497073] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10079.754346] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10079.754390] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.101@tcp (at 0@lo) [10079.754392] Lustre: Skipped 1 previous similar message [10080.224412] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [10082.237837] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 21:57:27 (1746151047) [10114.023138] Lustre: Failing over lustre-OST0000 [10114.046214] LustreError: 8341:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10114.051333] LustreError: 8341:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10114.082945] Lustre: server umount lustre-OST0000 complete [10114.820894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10114.823487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10114.823489] Lustre: Skipped 1 previous similar message [10114.826819] LustreError: 23661:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.851517] LustreError: Skipped 1 previous similar message [10118.957053] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10118.964663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10119.029990] Lustre: 8939:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [10119.175929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10119.188300] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10120.720816] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10121.201945] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10121.201950] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.101@tcp (at 0@lo) [10121.201951] Lustre: Skipped 1 previous similar message [10122.179930] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [10124.407743] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 21:58:09 (1746151089) [10139.899437] Lustre: Failing over lustre-OST0000 [10139.935775] LustreError: 10369:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10139.938708] LustreError: 10369:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10139.966471] Lustre: server umount lustre-OST0000 complete [10141.236762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10144.583673] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10144.589362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10144.660845] Lustre: 10983:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [10144.771195] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10144.787594] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10145.914613] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10145.997433] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10145.997521] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.101@tcp (at 0@lo) [10145.997523] Lustre: Skipped 1 previous similar message [10147.607198] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [10149.749205] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 21:58:35 (1746151115) [10156.262632] Lustre: Failing over lustre-OST0000 [10156.302178] LustreError: 11813:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10156.305818] LustreError: 11813:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10156.350740] Lustre: server umount lustre-OST0000 complete [10159.802925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10159.811939] Lustre: Skipped 3 previous similar messages [10166.797843] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10166.808046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10166.874510] Lustre: 12904:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [10166.997556] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10167.011767] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10168.159719] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10168.797349] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10168.800817] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.101@tcp (at 0@lo) [10168.800819] Lustre: Skipped 1 previous similar message [10169.577801] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [10171.301612] Lustre: DEBUG MARKER: == sanity test 165g: ofd_access_log_reader --keepalive works ========================================================== 21:58:57 (1746151137) [10213.092983] Lustre: Failing over lustre-OST0000 [10213.127485] LustreError: 13838:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10213.131125] LustreError: 13838:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10213.155477] Lustre: server umount lustre-OST0000 complete [10213.877915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10213.879420] LustreError: 5362:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10213.879422] LustreError: 5362:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 15 previous similar messages [10213.895991] LustreError: Skipped 2 previous similar messages [10221.452794] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10221.459009] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10221.522651] Lustre: 14991:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [10221.617439] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10221.627839] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10222.987070] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10223.002300] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.101@tcp (at 0@lo) [10223.002930] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10223.009528] Lustre: Skipped 1 previous similar message [10223.984314] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [10225.651886] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 21:59:51 (1746151191) [10226.462275] Lustre: DEBUG MARKER: creating a 10 Mb file [10261.498685] Lustre: DEBUG MARKER: starting reads [10262.371549] Lustre: DEBUG MARKER: truncating the file [10263.421633] Lustre: DEBUG MARKER: killing dd [10264.315942] Lustre: DEBUG MARKER: removing the temporary file [10268.151621] Lustre: DEBUG MARKER: == sanity test 170a: test lctl df to handle corrupted log ========================================================== 22:00:33 (1746151233) [10272.709060] Lustre: DEBUG MARKER: == sanity test 170b: check filename encoding ============= 22:00:38 (1746151238) [10285.147887] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 22:00:50 (1746151250) [10292.598987] Lustre: DEBUG MARKER: == sanity test 172: manual device removal with lctl cleanup/detach ================================================================ 22:00:58 (1746151258) [10297.815618] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 22:01:03 (1746151263) [10298.721852] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [10299.795860] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 22:01:05 (1746151265) [10301.485022] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_module obdecho/obdecho [10311.770869] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 22:01:17 (1746151277) [10313.310892] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_module obdecho/obdecho [10313.398606] Lustre: Echo OBD driver; http://www.lustre.org/ [10326.464309] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 22:01:31 (1746151291) [10380.885400] Lustre: DEBUG MARKER: == sanity test 182a: Test parallel modify metadata operations from mdc ========================================================== 22:02:26 (1746151346) [10406.914146] Lustre: DEBUG MARKER: == sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 22:02:52 (1746151372) [10772.522400] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 22:08:58 (1746151738) [10773.222946] Lustre: *** cfs_fail_loc=148, val=0*** [10777.183871] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 22:09:02 (1746151742) [10783.004511] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 22:09:08 (1746151748) [10787.528636] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 22:09:13 (1746151753) [10803.593370] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 22:09:29 (1746151769) [10809.027646] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 22:09:34 (1746151774) [10813.966875] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 22:09:39 (1746151779) [10817.653756] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 22:09:43 (1746151783) [10822.563486] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 22:09:48 (1746151788) [10830.746217] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 22:09:56 (1746151796) [10836.033004] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 22:10:01 (1746151801) [10840.200529] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 22:10:05 (1746151805) [10860.785222] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 22:10:26 (1746151826) [10864.643260] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 22:10:30 (1746151830) [10869.240614] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 22:10:34 (1746151834) [10872.812149] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 22:10:38 (1746151838) [10876.115243] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 22:10:42 (1746151842) [10879.840189] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 22:10:45 (1746151845) [10883.469814] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 22:10:49 (1746151849) [10887.188687] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 22:10:52 (1746151852) [10891.198773] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 22:10:56 (1746151856) [10895.768166] Lustre: lustre-MDD0000: changelog on [10895.770888] Lustre: Skipped 3 previous similar messages [10899.162305] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [10899.987942] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.7483 [10901.350611] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [10902.158266] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.5479 [10903.452124] Lustre: DEBUG MARKER: Test: lfs mkdir -i 1 /mnt/lustre/d205a.sanity.remote [10904.191314] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.11622 [10905.506407] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [10906.289805] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.4598 [10907.620469] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [10908.450376] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.24732 [10909.941790] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [10910.864585] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.24718 [10912.541855] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [10913.505554] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.30180 [10915.399581] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [10916.400574] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.29641 [10918.024484] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [10918.789417] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.10013 [10920.067467] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [10920.884885] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.22778 [10922.737920] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [10923.532450] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.27100 [10924.815336] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [10925.549431] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.15202 [10930.820003] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [10931.707001] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg301-client.v [10933.296694] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [10934.275020] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg301-client.E [10935.766218] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [10936.657217] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.oleg301-client.v [10941.255708] Lustre: lustre-MDD0001: changelog off [10941.258099] Lustre: Skipped 5 previous similar messages [10946.760320] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 22:11:52 (1746151912) [10952.858677] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 22:11:58 (1746151918) [10956.094716] Lustre: DEBUG MARKER: == sanity test 205d: verify the format of some stats files ========================================================== 22:12:01 (1746151921) [10965.818688] Lustre: DEBUG MARKER: == sanity test 205e: verify the output of lljobstat ====== 22:12:11 (1746151931) [10976.813206] Lustre: DEBUG MARKER: == sanity test 205f: verify qos_ost_weights YAML format == 22:12:22 (1746151942) [10981.956226] Lustre: DEBUG MARKER: == sanity test 205g: stress test for job_stats procfile == 22:12:27 (1746151947) [11077.562473] Lustre: DEBUG MARKER: == sanity test 205h: check jobid xattr is stored correctly ========================================================== 22:14:03 (1746152043) [11084.414312] Lustre: DEBUG MARKER: == sanity test 205i: check job_xattr parameter accepts and rejects values correctly ========================================================== 22:14:10 (1746152050) [11092.103947] Lustre: DEBUG MARKER: == sanity test 205k: Verify '?' operator on job stats ==== 22:14:17 (1746152057) [11097.541756] Lustre: DEBUG MARKER: == sanity test 205l: Verify job stats can scale ========== 22:14:23 (1746152063) [11178.962561] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 22:15:44 (1746152144) [11183.030513] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 22:15:48 (1746152148) [11187.423956] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 22:15:52 (1746152152) [11191.929830] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 22:15:57 (1746152157) [11199.414971] Lustre: Failing over lustre-MDT0000 [11199.600329] LustreError: 26310:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11199.603034] LustreError: 26310:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11199.759882] Lustre: server umount lustre-MDT0000 complete [11199.781021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11199.785398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11199.790482] Lustre: Skipped 3 previous similar messages [11199.809387] LustreError: 2743:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11199.820718] LustreError: 2743:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [11213.608671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11213.682213] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11213.829867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11213.873862] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11215.419814] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [11218.286223] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11218.848588] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.203.101@tcp (at 0@lo) [11218.853758] Lustre: Skipped 1 previous similar message [11218.871577] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11218.897409] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:38027 to 0x2c0000403:38049) [11218.901395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:36951 to 0x280000403:36993) [11220.863463] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11221.841331] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11226.704001] Lustre: Failing over lustre-MDT0000 [11226.807002] LustreError: 27602:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11226.809944] LustreError: 27602:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11226.872829] Lustre: server umount lustre-MDT0000 complete [11228.321018] LustreError: 4238:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11228.340665] LustreError: 4238:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 15 previous similar messages [11228.888463] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11228.901692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11228.921182] Lustre: Skipped 6 previous similar messages [11240.904255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11240.975596] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11241.169801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11241.214294] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11242.808176] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [11243.324953] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11246.174193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.203.101@tcp (at 0@lo) [11246.183786] Lustre: Skipped 3 previous similar messages [11246.227985] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11246.254730] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:38027 to 0x2c0000403:38081) [11246.255976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:36951 to 0x280000403:37025) [11248.095040] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11249.049337] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11254.642457] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 22:17:00 (1746152220) [11269.747731] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 22:17:15 (1746152235) [11275.356027] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 22:17:21 (1746152241) [11280.533606] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 22:17:26 (1746152246) [11294.269815] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 22:17:39 (1746152259) [11318.384726] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 22:18:03 (1746152283) [11322.277469] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 22:18:07 (1746152287) [11333.890591] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with embedded hyphen ('-') ========================================================== 22:18:19 (1746152299) [11338.962873] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 22:18:24 (1746152304) [11339.844161] Lustre: DEBUG MARKER: creating a 10 Mb file [11373.841361] Lustre: DEBUG MARKER: starting reads [11374.977277] Lustre: DEBUG MARKER: truncating the file [11376.281487] Lustre: DEBUG MARKER: killing dd [11377.349649] Lustre: DEBUG MARKER: removing the temporary file [11381.734415] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 22:19:07 (1746152347) [11386.356685] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 22:19:11 (1746152351) [11389.851908] Lustre: *** cfs_fail_loc=229, val=1*** [11391.540908] Lustre: *** cfs_fail_loc=229, val=1*** [11391.543325] Lustre: Skipped 1 previous similar message [11394.857652] Lustre: *** cfs_fail_loc=229, val=0*** [11394.859928] Lustre: Skipped 1 previous similar message [11407.424030] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 22:19:33 (1746152373) [11412.591067] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 22:19:38 (1746152378) [11417.465517] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 22:19:42 (1746152382) [11422.518571] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 22:19:48 (1746152388) [11427.711532] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 22:19:53 (1746152393) [11427.908747] Lustre: lustre-OST0001: Client 082bc5e7-9406-4ead-9c0c-a5402da9f4ab (at 192.168.203.1@tcp) reconnecting [11438.452977] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 22:20:03 (1746152403) [11441.805526] LustreError: 27230:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 sleeping for 3000ms [11444.814193] LustreError: 27230:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 awake [11451.474602] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 22:20:17 (1746152417) [11459.421821] Lustre: *** cfs_fail_loc=520, val=57344*** [11459.471705] LNet: *** cfs_fail_loc=e000, val=2147483648*** [11464.516050] Lustre: lustre-MDT0000: Client 082bc5e7-9406-4ead-9c0c-a5402da9f4ab (at 192.168.203.1@tcp) reconnecting [11464.541970] Lustre: 2741:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff880097e8aa00 x1830968639049856/t34359739958(0) o35->082bc5e7-9406-4ead-9c0c-a5402da9f4ab@192.168.203.1@tcp:0/0 lens 392/456 e 0 to 0 dl 1746152435 ref 1 fl Interpret:/202/0 rc 0/0 job:'dd.0' uid:0 gid:0 [11475.515390] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 22:20:41 (1746152441) [11477.998205] LustreError: 25526:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 sleeping for 22000ms [11498.028902] Lustre: lustre-OST0000: Client 082bc5e7-9406-4ead-9c0c-a5402da9f4ab (at 192.168.203.1@tcp) reconnecting [11499.654939] LustreError: 25526:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout interrupted [11499.659138] LustreError: 25526:0:(ldlm_lib.c:3490:target_bulk_io()) @@@ bulk READ failed: rc = -107 req@ffff880123e50380 x1830968639063168/t0(0) o3->082bc5e7-9406-4ead-9c0c-a5402da9f4ab@192.168.203.1@tcp:0/0 lens 488/440 e 0 to 0 dl 1746152463 ref 1 fl Interpret:/200/0 rc 0/0 job:'dd.0' uid:0 gid:0 [11499.675193] Lustre: 25526:0:(service.c:2494:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/2s); client may timeout req@ffff880123e50380 x1830968639063168/t0(0) o3->082bc5e7-9406-4ead-9c0c-a5402da9f4ab@192.168.203.1@tcp:0/0 lens 488/440 e 0 to 0 dl 1746152463 ref 1 fl Complete:/200/0 rc 0/0 job:'dd.0' uid:0 gid:0 [11504.588539] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [11505.719505] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [11506.631993] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 22:21:12 (1746152472) [11510.831496] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 22:21:16 (1746152476) [11515.261507] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 22:21:20 (1746152480) [11519.914482] Lustre: DEBUG MARKER: == sanity test 226d: verify fid2path with -n and -fn option ========================================================== 22:21:25 (1746152485) [11524.316378] Lustre: DEBUG MARKER: == sanity test 226e: Verify path2fid -0 option with newline and space ========================================================== 22:21:29 (1746152489) [11528.668914] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 22:21:34 (1746152494) [11532.395356] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 22:21:38 (1746152498) [11680.079684] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 22:24:05 (1746152645) [11798.184159] Lustre: Failing over lustre-MDT0000 [11799.132336] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11799.138183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11799.154819] Lustre: Skipped 3 previous similar messages [11800.642070] LustreError: 9256:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11800.647325] LustreError: 9256:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11800.731302] Lustre: server umount lustre-MDT0000 complete [11801.701075] LustreError: 4238:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.708669] LustreError: 4238:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 14 previous similar messages [11805.696622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11805.778328] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11805.966512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11806.011193] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11806.634566] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11807.595938] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [11810.967803] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.203.101@tcp (at 0@lo) [11810.974029] Lustre: Skipped 3 previous similar messages [11810.996582] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11811.023070] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:44260 to 0x2c0000403:44289) [11811.023667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:43238 to 0x280000403:43265) [11838.291700] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 22:26:43 (1746152803) [12090.599427] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 22:30:56 (1746153056) [12093.181237] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 22:30:59 (1746153059) [12096.343866] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 22:31:02 (1746153062) [12125.600994] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 22:31:31 (1746153091) [12126.703903] Lustre: *** cfs_fail_loc=1801, val=0*** [12126.710602] LustreError: 14279:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240001b77:0x2f6e:0x0]/f0 failed: rc = -5 [12127.528760] LustreError: 14279:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0000: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush migrate_dir' to finish migration: rc = -1 [12127.539293] LustreError: 14279:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200004281:0x56fa:0x0]/migrate_dir failed: rc = -1 [12129.610916] Lustre: 14279:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0001: [0x240001b71:0x16:0x0]/foo is open, migrate only dentry [12133.277602] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [12133.981842] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 22:31:39 (1746153099) [12137.953017] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 22:31:43 (1746153103) [12142.718636] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 22:31:48 (1746153108) [12145.748371] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 22:31:51 (1746153111) [12148.616409] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 22:31:54 (1746153114) [12152.119141] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 22:31:58 (1746153118) [12155.177347] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 22:32:01 (1746153121) [12155.876938] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [12156.554413] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 22:32:02 (1746153122) [12208.273061] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 22:32:54 (1746153174) [12213.826103] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 22:32:59 (1746153179) [12216.710828] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 22:33:02 (1746153182) [12231.549448] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 22:33:17 (1746153197) [12255.279639] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 22:33:41 (1746153221) [12277.821551] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 22:34:03 (1746153243) [12282.142387] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 22:34:07 (1746153247) [12286.738866] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 22:34:12 (1746153252) [12290.022473] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 22:34:16 (1746153256) [12290.605767] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [12291.295700] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 22:34:17 (1746153257) [12291.915673] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [12292.658922] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 22:34:18 (1746153258) [12296.594483] Lustre: DEBUG MARKER: == sanity test 230x: dir migration check space =========== 22:34:22 (1746153262) [12303.428491] LustreError: 2753:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000007:0x1:0x0]/d230x.sanity failed: rc = -28 [12303.433205] LustreError: 2753:0:(mdt_reint.c:2523:mdt_reint_migrate()) Skipped 1 previous similar message [12323.545150] Lustre: DEBUG MARKER: == sanity test 230y: unlink dir with bad hash type ======= 22:34:49 (1746153289) [12325.651389] Lustre: *** cfs_fail_loc=1802, val=0*** [12325.729297] LustreError: 14279:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240001b77:0x35cf:0x0]/d33 failed: rc = -77 [12332.084560] Lustre: DEBUG MARKER: == sanity test 230z: resume dir migration with bad hash type ========================================================== 22:34:58 (1746153298) [12334.255409] Lustre: *** cfs_fail_loc=1802, val=0*** [12334.256953] Lustre: Skipped 1 previous similar message [12334.310537] LustreError: 14279:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240001b77:0x3606:0x0]/d92 failed: rc = -77 [12356.108667] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 22:35:22 (1746153322) [12359.929558] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 22:35:25 (1746153325) [12383.802604] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 22:35:49 (1746153349) [12384.327011] Lustre: *** cfs_fail_loc=31c, val=0*** [12386.267644] Lustre: Failing over lustre-OST0000 [12386.345204] LustreError: 22654:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [12386.348290] LustreError: 22654:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [12386.391813] Lustre: server umount lustre-OST0000 complete [12386.758533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12386.764318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12386.773227] Lustre: Skipped 3 previous similar messages [12386.777330] LustreError: 23663:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12386.784560] LustreError: 23663:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [12389.936980] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [12389.943006] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12389.999863] Lustre: 23254:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [12390.122038] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12390.133707] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [12390.536129] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [12392.100349] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [12392.100476] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.101@tcp (at 0@lo) [12392.100477] Lustre: Skipped 3 previous similar messages [12392.174835] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [12395.667436] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 22:36:01 (1746153361) [12396.222360] Lustre: *** cfs_fail_loc=31c, val=0*** [12396.227231] LustreError: 5356:0:(ldlm_request.c:475:ldlm_cli_enqueue_local()) ### delayed lvb init failed (rc -12) ns: filter-lustre-OST0000_UUID lock: ffff88012eadde00/0x611abcaa87104913 lrc: 2/0,0 mode: --/PR res: [0x280000403:0xd62a:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 5356 timeout: 0 lvb_type: 0 [12397.769461] Lustre: Failing over lustre-OST0000 [12397.810224] Lustre: server umount lustre-OST0000 complete [12400.799677] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [12400.805534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12400.857731] Lustre: 24899:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [12400.940152] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12402.621348] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [12405.765133] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 22:36:11 (1746153371) [12408.665832] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 22:36:14 (1746153374) [12411.701582] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 22:36:17 (1746153377) [12414.834657] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 22:36:20 (1746153380) [12419.896384] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 22:36:25 (1746153385) [12423.425782] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 22:36:29 (1746153389) [12426.524071] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 22:36:32 (1746153392) [12469.685196] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 22:37:15 (1746153435) [12470.811785] Lustre: *** cfs_fail_loc=2100, val=0*** [12473.638195] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 22:37:19 (1746153439) [12478.384568] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 22:37:24 (1746153444) [12479.421731] Lustre: *** cfs_fail_loc=713, val=1*** [12479.423855] Lustre: Skipped 1 previous similar message [12484.979247] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 22:37:30 (1746153450) [12512.500101] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 22:37:58 (1746153478) [12524.275057] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 22:38:10 (1746153490) [12524.716550] Lustre: *** cfs_fail_loc=105, val=0*** [12526.957892] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 22:38:13 (1746153493) [12633.855399] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 22:39:59 (1746153599) [12687.900818] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 22:40:53 (1746153653) [12690.959167] Lustre: DEBUG MARKER: == sanity test 245a: check mdc connection flag/data: multiple modify RPCs ========================================================== 22:40:56 (1746153656) [12693.569529] Lustre: DEBUG MARKER: == sanity test 245b: check osp connection flag/data: multiple modify RPCs ========================================================== 22:40:59 (1746153659) [12696.998776] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 22:41:02 (1746153662) [12700.782434] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 22:41:06 (1746153666) [12703.751199] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 22:41:09 (1746153669) [12707.547825] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 22:41:13 (1746153673) [12711.370657] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 22:41:17 (1746153677) [12714.387227] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 22:41:20 (1746153680) [12715.102292] Lustre: enable_remote_subdir_mount is deprecated, it's always enabled. [12721.532908] Lustre: DEBUG MARKER: == sanity test 247g: striped directory submount revalidate ROOT from cache ========================================================== 22:41:27 (1746153687) [12726.601703] Lustre: DEBUG MARKER: == sanity test 247h: remote directory submount revalidate ROOT from cache ========================================================== 22:41:32 (1746153692) [12734.588457] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 22:41:40 (1746153700) [12792.096017] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 22:42:37 (1746153757) [12819.370235] Lustre: DEBUG MARKER: == sanity test 248c: verify whole file read behavior ===== 22:43:05 (1746153785) [12830.530968] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 22:43:16 (1746153796) [12834.132454] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 22:43:19 (1746153799) [12837.660286] Lustre: DEBUG MARKER: == sanity test 251a: Handling short read and write correctly ========================================================== 22:43:23 (1746153803) [12841.852932] Lustre: DEBUG MARKER: == sanity test 251b: short read restore offset correctly ========================================================== 22:43:27 (1746153807) [12849.363999] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 22:43:35 (1746153815) [12853.618031] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 22:43:39 (1746153819) [12886.020937] LustreError: 4232:0:(lod_qos.c:1380:lod_ost_alloc_specific()) can't lstripe objid [0x200004284:0x40:0x0]: have 0 want 1 [12925.403900] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 22:44:51 (1746153891) [12927.296452] Lustre: lustre-MDD0000: changelog on [12927.301219] Lustre: Skipped 1 previous similar message [12933.066405] Lustre: lustre-MDD0001: changelog off [12933.067503] Lustre: Skipped 1 previous similar message [12935.656830] Lustre: DEBUG MARKER: SKIP: sanity test_255a skipping excluded test 255a (base 255) [12936.470368] Lustre: DEBUG MARKER: SKIP: sanity test_255b skipping excluded test 255b (base 255) [12937.243751] Lustre: DEBUG MARKER: SKIP: sanity test_255c skipping excluded test 255c (base 255) [12938.129193] Lustre: DEBUG MARKER: SKIP: sanity test_256 skipping excluded test 256 [12938.913988] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 22:45:04 (1746153904) [12939.705374] Lustre: *** cfs_fail_loc=161, val=0*** [12940.433449] Lustre: Failing over lustre-MDT0001 [12942.774064] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12942.777370] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12942.782992] Lustre: Skipped 3 previous similar messages [12942.785809] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12942.789821] Lustre: Skipped 3 previous similar messages [12944.821237] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12944.824012] Lustre: Skipped 2 previous similar messages [12945.685445] LustreError: 8881:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [12945.688078] LustreError: 8881:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [12945.763949] Lustre: server umount lustre-MDT0001 complete [12948.537833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12948.593747] Lustre: 9479:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [12948.732858] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12948.752438] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [12948.755251] Lustre: Skipped 1 previous similar message [12949.743506] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [12949.900421] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [12949.903566] Lustre: Skipped 1 previous similar message [12951.122246] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [12953.734611] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.203.101@tcp (at 0@lo) [12953.738490] Lustre: Skipped 3 previous similar messages [12953.749286] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [12953.754057] Lustre: Skipped 1 previous similar message [12953.779784] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:23570 to 0x280000400:23585) [12953.787630] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:23641 to 0x2c0000400:23681) [12958.439218] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 22:45:24 (1746153924) [12960.820888] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 22:45:26 (1746153926) [12963.616328] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 22:45:29 (1746153929) [12976.832122] Lustre: *** cfs_fail_loc=2301, val=0*** [12977.949200] Lustre: Failing over lustre-OST0000 [12977.991139] Lustre: server umount lustre-OST0000 complete [12978.774250] LustreError: 5359:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12978.780796] LustreError: 5359:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [12981.324645] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [12981.331586] LDISKFS-fs (dm-2): 1 truncate cleaned up [12981.333333] LDISKFS-fs (dm-2): recovery complete [12981.341309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12981.395515] Lustre: 12722:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [12981.486457] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12983.286594] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [12988.422513] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 22:45:54 (1746153954) [12990.863060] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 22:45:56 (1746153956) [12996.730663] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 22:46:02 (1746153962) [12999.114462] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 22:46:05 (1746153965) [13001.317435] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 22:46:07 (1746153967) [13003.699518] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 22:46:09 (1746153969) [13006.918010] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 22:46:12 (1746153972) [13009.653684] Lustre: Increasing provided stripe size to a minimum value 64 [13014.059890] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 22:46:20 (1746153980) [13016.345668] Lustre: *** cfs_fail_loc=168, val=40*** [13017.005153] Lustre: *** cfs_fail_loc=168, val=20*** [13017.007251] Lustre: Skipped 2 previous similar messages [13018.381622] Lustre: *** cfs_fail_loc=168, val=15*** [13018.383449] Lustre: Skipped 5 previous similar messages [13023.508530] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 22:46:29 (1746153989) [13027.876879] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 22:46:33 (1746153993) [13030.831928] Lustre: DEBUG MARKER: == sanity test 270j: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 22:46:36 (1746153996) [13034.377402] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 22:46:40 (1746154000) [13036.832032] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 22:46:42 (1746154002) [13039.162718] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 22:46:45 (1746154005) [13041.413704] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 22:46:47 (1746154007) [13104.852382] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 22:47:50 (1746154070) [13108.267369] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 22:47:54 (1746154074) [13111.386614] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 22:47:57 (1746154077) [13115.314550] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 22:48:01 (1746154081) [13118.756419] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 22:48:04 (1746154084) [13123.196886] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 22:48:09 (1746154089) [13126.672646] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 22:48:12 (1746154092) [13130.121471] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 22:48:16 (1746154096) [13134.263969] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 22:48:20 (1746154100) [13137.983819] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 22:48:23 (1746154103) [13141.012869] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 22:48:26 (1746154106) [13141.933278] LustreError: 2752:0:(mdt_io.c:571:mdt_commitrw_write()) cfs_fail_timeout id 16b sleeping for 2000ms [13143.937196] LustreError: 2752:0:(mdt_io.c:571:mdt_commitrw_write()) cfs_fail_timeout id 16b awake [13146.413530] Lustre: DEBUG MARKER: == sanity test 273c: race writeback and object destroy === 22:48:32 (1746154112) [13149.571256] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 22:48:35 (1746154115) [13153.832716] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 22:48:39 (1746154119) [13154.601547] Lustre: Failing over lustre-OST0000 [13154.603490] LustreError: 25412:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13154.659360] LustreError: 25407:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13154.662593] LustreError: 25407:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [13154.748381] Lustre: server umount lustre-OST0000 complete [13156.758665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13156.767727] Lustre: Skipped 4 previous similar messages [13157.816082] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13157.823950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13157.890375] Lustre: 26681:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [13157.925964] LustreError: 26715:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13157.931211] LustreError: 26715:0:(obd_class.h:479:obd_check_dev()) Skipped 2 previous similar messages [13158.009335] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13158.018569] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13158.023610] Lustre: Skipped 1 previous similar message [13159.276820] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13159.281454] Lustre: Skipped 1 previous similar message [13159.643303] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13159.646795] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.101@tcp (at 0@lo) [13159.646797] Lustre: Skipped 4 previous similar messages [13159.662742] Lustre: Skipped 1 previous similar message [13159.917335] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13161.423469] Lustre: Failing over lustre-OST0000 [13161.427930] LustreError: 28041:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13161.483238] Lustre: server umount lustre-OST0000 complete [13164.743523] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13164.748370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13166.886703] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13168.483496] Lustre: Failing over lustre-OST0000 [13168.563245] Lustre: server umount lustre-OST0000 complete [13171.669063] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13171.674998] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13171.864298] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13171.881094] Lustre: Skipped 1 previous similar message [13173.828983] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13175.212602] Lustre: Failing over lustre-OST0000 [13175.214598] LustreError: 796:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13175.271448] Lustre: server umount lustre-OST0000 complete [13178.324446] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13178.336849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13180.159130] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13181.612780] Lustre: Failing over lustre-OST0000 [13181.614721] LustreError: 3476:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13181.614723] LustreError: 3476:0:(obd_class.h:479:obd_check_dev()) Skipped 2 previous similar messages [13181.706346] Lustre: server umount lustre-OST0000 complete [13185.069412] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13185.075874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13186.992267] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13188.380523] Lustre: Failing over lustre-OST0000 [13188.445604] Lustre: server umount lustre-OST0000 complete [13191.570268] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13191.578301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13191.724163] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13191.733627] Lustre: Skipped 2 previous similar messages [13193.874418] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13195.284920] Lustre: Failing over lustre-OST0000 [13195.288044] LustreError: 8677:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13195.291795] LustreError: 8677:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13195.352269] Lustre: server umount lustre-OST0000 complete [13198.992353] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13198.999916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13201.189753] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13202.556058] Lustre: Failing over lustre-OST0000 [13202.635250] Lustre: server umount lustre-OST0000 complete [13205.799836] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13205.807279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13208.031159] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13209.728745] Lustre: Failing over lustre-OST0000 [13209.868786] Lustre: server umount lustre-OST0000 complete [13213.358077] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13213.365296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13213.442302] LustreError: 15177:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13213.445144] LustreError: 15177:0:(obd_class.h:479:obd_check_dev()) Skipped 4 previous similar messages [13215.640972] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13217.277047] Lustre: Failing over lustre-OST0000 [13217.373247] Lustre: server umount lustre-OST0000 complete [13220.677865] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13220.687366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13222.659127] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13224.056226] Lustre: Failing over lustre-OST0000 [13224.132588] Lustre: server umount lustre-OST0000 complete [13227.309648] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13227.320787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13227.392252] Lustre: 20264:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.101@tcp: no remote llog for lustre-sptlrpc, check MGS config [13227.397105] Lustre: 20264:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 9 previous similar messages [13227.514093] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13227.520561] Lustre: Skipped 4 previous similar messages [13229.297944] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13230.758247] Lustre: Failing over lustre-OST0000 [13230.825972] Lustre: server umount lustre-OST0000 complete [13233.731736] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13233.739531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13235.306069] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13236.660077] Lustre: Failing over lustre-OST0000 [13236.712402] Lustre: server umount lustre-OST0000 complete [13238.870865] LustreError: 15058:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13238.877412] LustreError: 15058:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 35 previous similar messages [13239.551170] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13239.556200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13241.347099] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13242.479114] Lustre: Failing over lustre-OST0000 [13242.559047] Lustre: server umount lustre-OST0000 complete [13245.663886] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13245.670670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13245.742914] LustreError: 28188:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13245.745354] LustreError: 28188:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [13247.592181] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13248.911162] Lustre: Failing over lustre-OST0000 [13248.963142] Lustre: server umount lustre-OST0000 complete [13252.152847] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13252.161462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13254.140462] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13255.365207] Lustre: Failing over lustre-OST0000 [13255.423276] Lustre: server umount lustre-OST0000 complete [13258.269344] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13258.279338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13260.360827] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13261.675416] Lustre: Failing over lustre-OST0000 [13261.737928] Lustre: server umount lustre-OST0000 complete [13264.546256] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13264.552892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13266.439832] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13267.682313] Lustre: Failing over lustre-OST0000 [13267.732193] Lustre: server umount lustre-OST0000 complete [13270.530902] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13270.537670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13272.286253] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13273.621844] Lustre: Failing over lustre-OST0000 [13273.678158] Lustre: server umount lustre-OST0000 complete [13276.476720] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13276.485148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13278.391056] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13279.876332] Lustre: Failing over lustre-OST0000 [13279.936417] Lustre: server umount lustre-OST0000 complete [13283.451638] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13283.459802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13285.357261] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13288.996458] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 22:50:54 (1746154254) [13292.138536] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 22:50:58 (1746154258) [13293.176265] Lustre: Failing over lustre-MDT0000 [13293.198204] Lustre: Failing over lustre-MDT0001 [13293.304458] LustreError: 13539:0:(tgt_mount.c:1756:server_put_super()) cfs_race id 60c sleeping [13293.473835] Lustre: server umount lustre-MDT0000 complete [13296.452970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13296.524192] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13296.532272] LustreError: 14119:0:(tgt_mount.c:1486:server_start_targets()) cfs_fail_race id 60c waking [13296.535830] LustreError: 13539:0:(tgt_mount.c:1756:server_put_super()) cfs_fail_race id 60c awake: rc=1772 [13296.557279] Lustre: server umount lustre-MDT0001 complete [13298.702286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13300.045820] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13304.508482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13304.706367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [13304.836014] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13305.578499] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:23602 to 0x280000400:23617) [13305.579170] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:23687 to 0x2c0000400:23713) [13306.318557] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13308.045255] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [13309.907576] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:55886 to 0x2c0000403:55905) [13309.908337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:54870 to 0x280000403:54913) [13310.936908] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 22:51:16 (1746154276) [13311.928645] LustreError: 14125:0:(llog_osd.c:1480:llog_osd_open()) cfs_race id 15e sleeping [13313.650152] Lustre: Failing over lustre-MDT0000 [13313.758394] LustreError: 16266:0:(tgt_mount.c:128:server_deregister_mount()) cfs_fail_race id 15e waking [13313.762049] LustreError: 14125:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e awake: rc=3172 [13314.776867] LustreError: 16266:0:(tgt_mount.c:128:server_deregister_mount()) cfs_fail_race id 15e waking [13314.800197] Lustre: server umount lustre-MDT0000 complete [13318.269576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13318.359171] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13318.389962] LustreError: 16870:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13319.472592] LustreError: 16870:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13320.480730] LustreError: 16870:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13322.492225] LustreError: 16870:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13322.495919] LustreError: 16870:0:(llog_osd.c:1480:llog_osd_open()) Skipped 1 previous similar message [13326.535338] LustreError: 16870:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13326.551313] LustreError: 16870:0:(llog_osd.c:1480:llog_osd_open()) Skipped 3 previous similar messages [13330.572587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13331.584129] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:55886 to 0x2c0000403:55937) [13331.584567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:54870 to 0x280000403:54945) [13331.794953] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug all all [13334.582254] LustreError: 16870:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13334.586007] LustreError: 16870:0:(llog_osd.c:1480:llog_osd_open()) Skipped 5 previous similar messages [13342.314501] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 22:51:48 (1746154308) [13346.013524] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 22:51:51 (1746154311) [13370.496483] Lustre: DEBUG MARKER: == sanity test 300c: chown [13526.186828] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 22:54:52 (1746154492) [13529.973878] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ========================================================== 22:54:55 (1746154495) [13534.074487] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ========================================================== 22:54:59 (1746154499) [13538.493121] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ========================================================== 22:55:04 (1746154504) [13546.453955] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory ========================================================== 22:55:12 (1746154512) [13553.629444] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory ========================================================== 22:55:19 (1746154519) [13589.581385] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ============ 22:55:55 (1746154555) [13592.991589] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ======== 22:55:58 (1746154558) [13596.736105] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ========================================================== 22:56:02 (1746154562) [13602.007238] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS ========================================================== 22:56:07 (1746154567) [13602.710856] Lustre: DEBUG MARKER: SKIP: sanity test_300m Only for single MDT [13603.616172] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA ========================================================== 22:56:09 (1746154569) [13612.386165] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [13613.206168] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================================== 22:56:19 (1746154579) [13613.823476] Lustre: *** cfs_fail_loc=1704, val=0*** [13613.825937] Lustre: 14556:0:(out_handler.c:911:out_tx_end()) lustre-MDT0001-osd: error during execution of #8 from /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:670: rc = -28 [13613.844778] LustreError: 14556:0:(out_handler.c:921:out_tx_end()) lustre-MDT0001-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:425: rc = -524 [13617.066993] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ========================================================== 22:56:22 (1746154582) [13620.656702] Lustre: DEBUG MARKER: == sanity test 300r: test -1 striped directory =========== 22:56:26 (1746154586) [13624.379513] Lustre: DEBUG MARKER: == sanity test 300s: test lfs mkdir -c without -i ======== 22:56:30 (1746154590) [13627.869068] Lustre: DEBUG MARKER: == sanity test 300t: test max_mdt_stripecount ============ 22:56:33 (1746154593) [13633.913758] Lustre: DEBUG MARKER: == sanity test 300ua: basic overstriped dir sanity test == 22:56:39 (1746154599) [13639.604299] Lustre: DEBUG MARKER: == sanity test 300ub: test MDT overstriping interface [13644.301311] Lustre: DEBUG MARKER: == sanity test 300uc: test MDT overstriping as default [13648.595139] Lustre: DEBUG MARKER: == sanity test 300ud: dir split ========================== 22:56:54 (1746154614) [13738.242499] Lustre: DEBUG MARKER: == sanity test 300ue: dir merge ========================== 22:58:24 (1746154704) [13783.104486] Lustre: DEBUG MARKER: == sanity test 300uf: migrate with too many local locks == 22:59:09 (1746154749) [13786.686745] Lustre: DEBUG MARKER: == sanity test 300ug: migrate overstriped dirs =========== 22:59:12 (1746154752) [13787.286723] Lustre: 2752:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 3793 > trans_max 3200 [13787.290970] Lustre: 2752:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 6/24/0, destroy: 6/24/0 [13787.296101] Lustre: 2752:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 31/2640/0 [13787.300761] Lustre: 2752:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 0/0/0 [13787.304316] Lustre: 2752:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 47/938/0, delete: 32/50/0 [13787.307563] Lustre: 2752:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 30/30/0, ref_del: 28/28/0 [13787.313502] CPU: 2 PID: 2752 Comm: mdt_io00_001 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [13787.317896] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [13787.322370] Call Trace: [13787.323703] [] dump_stack+0x19/0x1b [13787.325741] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [13787.329327] [] ? __llog_ctxt_put+0xd3/0x140 [obdclass] [13787.332441] [] top_trans_start+0x351/0xab0 [ptlrpc] [13787.335154] [] lod_trans_start+0x7f/0x2e0 [lod] [13787.337572] [] mdd_trans_start+0x14/0x20 [mdd] [13787.340651] [] mdd_migrate_object+0x12ee/0x2470 [mdd] [13787.344013] [] ? _raw_spin_unlock+0xe/0x20 [13787.346550] [] ? __mdt_stripe_get+0xe1/0x4e0 [mdt] [13787.349641] [] mdd_migrate+0x28/0x30 [mdd] [13787.352050] [] mdt_reint_migrate+0x1b9f/0x24b0 [mdt] [13787.356638] [] mdt_reint_rec+0x87/0x240 [mdt] [13787.359679] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [13787.362695] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [13787.366390] [] mdt_reint+0x67/0x150 [mdt] [13787.369061] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [13787.371849] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [13787.374476] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [13787.378914] [] ? put_prev_entity+0x31/0x400 [13787.386752] [] ? do_raw_spin_unlock+0x49/0x90 [13787.389587] [] ? _raw_spin_unlock_irq+0xe/0x30 [13787.391696] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [13787.394310] [] kthread+0xe4/0xf0 [13787.396528] [] ? kthread_create_on_node+0x140/0x140 [13787.399305] [] ret_from_fork_nospec_begin+0x7/0x21 [13787.403277] [] ? kthread_create_on_node+0x140/0x140 [13790.895232] Lustre: DEBUG MARKER: == sanity test 300uh: overstripe tunable max_stripes_per_mdt ========================================================== 22:59:16 (1746154756) [13795.000498] Lustre: DEBUG MARKER: == sanity test 300ui: overstripe is not supported on one MDT system ========================================================== 22:59:20 (1746154760) [13795.621893] Lustre: DEBUG MARKER: SKIP: sanity test_300ui 1 MDT only [13796.304937] Lustre: DEBUG MARKER: == sanity test 300uj: overstriped dir with -C -N sanity test ========================================================== 22:59:22 (1746154762) [13799.845344] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ============= 22:59:25 (1746154765) [13803.205302] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open ========================================================== 22:59:29 (1746154769) [13805.879109] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ========================================================== 22:59:31 (1746154771) [13806.482627] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [13807.255078] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ========================================================== 22:59:33 (1746154773) [13830.663502] Lustre: DEBUG MARKER: == sanity test 312: make sure ZFS adjusts its block size by write pattern ========================================================== 22:59:56 (1746154796) [13831.414454] Lustre: DEBUG MARKER: SKIP: sanity test_312 the test only applies to zfs [13832.112228] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ========================================================== 22:59:58 (1746154798) [13832.627672] Lustre: *** cfs_fail_loc=720, val=0*** [13832.630129] LustreError: 30936:0:(osd_handler.c:2213:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [13832.634957] LustreError: 30936:0:(ofd_objects.c:1014:ofd_object_punch()) lustre-OST0000: failed to stop transaction: rc = -5 [13835.505057] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ========================================================== 23:00:01 (1746154801) [13836.644227] Lustre: *** cfs_fail_loc=720, val=0*** [13836.645343] LustreError: 5362:0:(osd_handler.c:2213:osd_trans_stop()) lustre-OST0001: failed in transaction hook: rc = -5 [13836.645356] LustreError: 5362:0:(ofd_objects.c:1081:ofd_destroy()) lustre-OST0001 failed to stop transaction: -5 [13836.645425] LustreError: 5362:0:(ofd_dev.c:1765:ofd_destroy_hdl()) lustre-OST0001: error destroying object [0x2c0000403:0xdfb2:0x0]: -5 [13836.658955] Lustre: Skipped 1 previous similar message [13842.384417] LustreError: 16879:0:(osp_sync.c:1155:osp_sync_process_committed()) lustre-OST0000-osc-MDT0000: can't cancel record: rc = -2 [13842.394814] LustreError: 16879:0:(osp_sync.c:1155:osp_sync_process_committed()) Skipped 1 previous similar message [13847.390203] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ============= 23:00:13 (1746154813) [13855.594853] Lustre: DEBUG MARKER: == sanity test 316: lfs migrate of file with large_xattr enabled ========================================================== 23:00:21 (1746154821) [13859.870576] Lustre: DEBUG MARKER: == sanity test 317: Verify blocks get correctly update after truncate ========================================================== 23:00:25 (1746154825) [13864.044696] Lustre: DEBUG MARKER: == sanity test 318: Verify async readahead tunables ====== 23:00:29 (1746154829) [13867.564024] Lustre: DEBUG MARKER: == sanity test 319: lost lease lock on migrate error ===== 23:00:33 (1746154833) [13872.862638] Lustre: 2753:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0000: [0x200005224:0x142:0x0]/xxx is open, migrate only dentry [13876.327702] Lustre: DEBUG MARKER: == sanity test 350: force NID mismatch path to be exercised ========================================================== 23:00:42 (1746154842) [13877.038727] LNet: *** cfs_fail_loc=e001, val=100*** [13877.042314] LNet: 14556:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-0@lo matched 1830968578382336 with NID mismatch 0@lo accepted (same peer ffff88013736d000) [13877.571472] LNet: *** cfs_fail_loc=e001, val=100*** [13877.573503] LNet: Skipped 44 previous similar messages [13877.574801] LNet: 10547:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-192.168.203.1@tcp matched 1830968578388224 with NID mismatch 192.168.203.1@tcp accepted (same peer ffff880131345c00) [13877.581323] LNet: 10547:0:(lib-ptl.c:160:lnet_try_match_md()) Skipped 44 previous similar messages [14030.681400] Lustre: DEBUG MARKER: == sanity test 360: ldiskfs unlink in a separate thread == 23:03:16 (1746154996) [14042.802478] Lustre: DEBUG MARKER: == sanity test 398a: direct IO should cancel lock otherwise lockless ========================================================== 23:03:28 (1746155008) [14045.908982] Lustre: DEBUG MARKER: == sanity test 398b: DIO and buffer IO race ============== 23:03:31 (1746155011) [14125.397728] Lustre: DEBUG MARKER: == sanity test 398c: run fio to test AIO ================= 23:04:50 (1746155090) [14162.098918] Lustre: DEBUG MARKER: == sanity test 398d: run aiocp to verify block size > stripe size ========================================================== 23:05:27 (1746155127) [14177.551707] Lustre: DEBUG MARKER: == sanity test 398e: O_Direct open cleared by fcntl doesn't cause hang ========================================================== 23:05:43 (1746155143) [14181.419309] Lustre: DEBUG MARKER: == sanity test 398f: verify aio handles ll_direct_rw_pages errors correctly ========================================================== 23:05:47 (1746155147) [14188.955130] Lustre: DEBUG MARKER: == sanity test 398g: verify parallel dio async RPC submission ========================================================== 23:05:54 (1746155154) [14189.839520] LustreError: 1936:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14189.847051] LustreError: 1936:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [14191.849825] LustreError: 3866:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14192.206351] LustreError: 3869:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14192.209859] LustreError: 3869:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 4 previous similar messages [14194.219250] LustreError: 3869:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14194.224154] LustreError: 3869:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 7 previous similar messages [14194.285521] LustreError: 3869:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14196.297386] LustreError: 3869:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14196.338792] LustreError: 3869:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14198.347255] LustreError: 3869:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14200.429846] LustreError: 3869:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14200.436683] LustreError: 3869:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [14202.452248] LustreError: 3869:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14202.457301] LustreError: 3869:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [14209.451241] LustreError: 1936:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout interrupted [14212.084008] Lustre: DEBUG MARKER: == sanity test 398h: verify correctness of read [14224.031140] Lustre: DEBUG MARKER: == sanity test 398i: verify parallel dio handles ll_direct_rw_pages errors correctly ========================================================== 23:06:29 (1746155189) [14228.545693] Lustre: DEBUG MARKER: == sanity test 398j: test parallel dio where stripe size > rpc_size ========================================================== 23:06:34 (1746155194) [14242.603393] Lustre: DEBUG MARKER: == sanity test 398k: test enospc on first stripe ========= 23:06:48 (1746155208) [14256.768174] Lustre: DEBUG MARKER: SKIP: sanity test_398k 7196000 > 600000 skipping out-of-space test on OST0 [14257.749531] Lustre: DEBUG MARKER: == sanity test 398l: test enospc on intermediate stripe/RPC ========================================================== 23:07:03 (1746155223) [14263.567587] Lustre: DEBUG MARKER: SKIP: sanity test_398l 7176088 > 600000 skipping out-of-space test on OST0 [14278.235434] Lustre: DEBUG MARKER: == sanity test 398m: test RPC failures with parallel dio ========================================================== 23:07:24 (1746155244) [14278.913339] Lustre: *** cfs_fail_loc=20e, val=1*** [14279.970304] Lustre: *** cfs_fail_loc=20e, val=1*** [14279.973798] Lustre: Skipped 10 previous similar messages [14281.981458] Lustre: *** cfs_fail_loc=20e, val=1*** [14281.987178] Lustre: Skipped 1 previous similar message [14284.688687] Lustre: *** cfs_fail_loc=20e, val=1*** [14284.692884] Lustre: Skipped 5 previous similar messages [14288.699680] Lustre: *** cfs_fail_loc=20e, val=1*** [14288.702112] Lustre: Skipped 1 previous similar message [14299.744242] Lustre: *** cfs_fail_loc=20e, val=1*** [14299.746975] Lustre: Skipped 9 previous similar messages [14323.739826] Lustre: *** cfs_fail_loc=20e, val=1*** [14323.743412] Lustre: Skipped 11 previous similar messages [14356.792293] Lustre: *** cfs_fail_loc=20f, val=1*** [14356.800010] Lustre: Skipped 33 previous similar messages [14427.520330] Lustre: *** cfs_fail_loc=20e, val=2*** [14427.524754] Lustre: Skipped 54 previous similar messages [14509.252439] Lustre: DEBUG MARKER: == sanity test 398n: test append with parallel DIO ======= 23:11:14 (1746155474) [14521.752161] Lustre: DEBUG MARKER: == sanity test 398o: right kms with DIO ================== 23:11:27 (1746155487) [14522.842537] Lustre: DEBUG MARKER: sanity test_398o: @@@@@@ FAIL: bad KMS [14528.523709] Lustre: DEBUG MARKER: == sanity test 398p: race aio with buffered i/o ========== 23:11:34 (1746155494) [14568.012846] Lustre: DEBUG MARKER: == sanity test 398q: race dio with buffered i/o ========== 23:12:13 (1746155533) [14663.052594] Lustre: DEBUG MARKER: == sanity test 398r: i/o error on file read ============== 23:13:48 (1746155628) [14663.839782] Lustre: *** cfs_fail_loc=20f, val=0*** [14663.842359] Lustre: Skipped 56 previous similar messages [14723.426346] Lustre: DEBUG MARKER: == sanity test 398s: i/o error on mirror file read ======= 23:14:49 (1746155689) [14724.553934] Lustre: DEBUG MARKER: sanity test_398s: @@@@@@ FAIL: mirror resync failed