[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468426338 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002333] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007256] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010126] LSM: Security Framework initializing [ 0.011043] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013056] *** VALIDATE selinux *** [ 0.020515] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024757] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026083] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027093] *** VALIDATE tmpfs *** [ 0.028345] *** VALIDATE proc *** [ 0.029193] *** VALIDATE cgroup *** [ 0.030007] *** VALIDATE cgroup2 *** [ 0.031244] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034030] Spectre V2 : User space: Vulnerable [ 0.035008] Speculative Store Bypass: Vulnerable [ 0.037949] debug: unmapping init [mem 0xffffffffb5059000-0xffffffffb5060fff] [ 0.039890] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040666] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041021] ... version: 2 [ 0.042012] ... bit width: 48 [ 0.043011] ... generic registers: 4 [ 0.044011] ... value mask: 0000ffffffffffff [ 0.045011] ... max period: 00007fffffffffff [ 0.046014] ... fixed-purpose events: 3 [ 0.047012] ... event mask: 000000070000000f [ 0.048299] rcu: Hierarchical SRCU implementation. [ 0.050402] smp: Bringing up secondary CPUs ... [ 0.051559] x86: Booting SMP configuration: [ 0.052024] .... node #0, CPUs: #1 #2 #3 [ 0.056147] smp: Brought up 1 node, 4 CPUs [ 0.058015] smpboot: Max logical packages: 1 [ 0.059013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146022] node 0 deferred pages initialised in 84ms [ 0.149214] devtmpfs: initialized [ 0.150221] x86/mm: Memory block size: 128MB [ 0.152000] gcov: version magic: 0x41383552 [ 0.154344] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155060] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157235] pinctrl core: initialized pinctrl subsystem [ 0.158157] [ 0.158448] ************************************************************* [ 0.161009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163012] ** ** [ 0.165014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168033] ** ** [ 0.169007] ** This means that this kernel is built to expose internal ** [ 0.170009] ** IOMMU data structures, which may compromise security on ** [ 0.172010] ** your system. ** [ 0.174011] ** ** [ 0.176043] ** If you see this message and you are not debugging the ** [ 0.179014] ** kernel, report this immediately to your vendor! ** [ 0.181009] ** ** [ 0.182011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185017] ************************************************************* [ 0.187498] NET: Registered protocol family 16 [ 0.188357] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198461] cpuidle: using governor menu [ 0.200591] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202435] PCI: Using configuration type 1 for base access [ 0.204109] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214045] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.217188] cryptd: max_cpu_qlen set to 1000 [ 0.220239] ACPI: Added _OSI(Module Device) [ 0.222016] ACPI: Added _OSI(Processor Device) [ 0.223014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.225010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.229122] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235442] ACPI: Interpreter enabled [ 0.237056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238012] ACPI: Using IOAPIC for interrupt routing [ 0.240120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243420] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253223] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.256053] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259112] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.263129] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269000] acpiphp: Slot [2] registered [ 0.271215] acpiphp: Slot [5] registered [ 0.273195] acpiphp: Slot [6] registered [ 0.275196] acpiphp: Slot [7] registered [ 0.276217] acpiphp: Slot [8] registered [ 0.278285] acpiphp: Slot [9] registered [ 0.280113] acpiphp: Slot [10] registered [ 0.284145] acpiphp: Slot [3] registered [ 0.286101] acpiphp: Slot [4] registered [ 0.287090] acpiphp: Slot [11] registered [ 0.288122] acpiphp: Slot [12] registered [ 0.290097] acpiphp: Slot [13] registered [ 0.291120] acpiphp: Slot [14] registered [ 0.293100] acpiphp: Slot [15] registered [ 0.294099] acpiphp: Slot [16] registered [ 0.296106] acpiphp: Slot [17] registered [ 0.297117] acpiphp: Slot [18] registered [ 0.299110] acpiphp: Slot [19] registered [ 0.300095] acpiphp: Slot [20] registered [ 0.302126] acpiphp: Slot [21] registered [ 0.304107] acpiphp: Slot [22] registered [ 0.305071] acpiphp: Slot [23] registered [ 0.306103] acpiphp: Slot [24] registered [ 0.308166] acpiphp: Slot [25] registered [ 0.310118] acpiphp: Slot [26] registered [ 0.311099] acpiphp: Slot [27] registered [ 0.313105] acpiphp: Slot [28] registered [ 0.314132] acpiphp: Slot [29] registered [ 0.316142] acpiphp: Slot [30] registered [ 0.318141] acpiphp: Slot [31] registered [ 0.320086] PCI host bridge to bus 0000:00 [ 0.321031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.324050] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.327035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.330037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.336034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338216] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342115] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.346301] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.366049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.368031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.369011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.376516] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.379716] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.385153] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.390020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.405020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.409021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.415531] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.421015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.426013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.445016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.457165] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.471022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.481022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.501020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.517461] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.527021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.535021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.552019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.562963] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.573017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.583017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.604018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.615457] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.624020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.636021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.659021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.670681] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.679017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.686019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.700023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.711000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.714459] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.716447] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.719451] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.723330] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.728134] iommu: Default domain type: Passthrough [ 0.730407] SCSI subsystem initialized [ 0.732254] ACPI: bus type USB registered [ 0.734108] usbcore: registered new interface driver usbfs [ 0.736104] usbcore: registered new interface driver hub [ 0.738091] usbcore: registered new device driver usb [ 0.739127] pps_core: LinuxPPS API ver. 1 registered [ 0.741015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.745064] PTP clock support registered [ 0.747044] EDAC MC: Ver: 3.0.0 [ 0.749144] PCI: Using ACPI for IRQ routing [ 0.751325] NetLabel: Initializing [ 0.753014] NetLabel: domain hash size = 128 [ 0.755016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.757085] NetLabel: unlabeled traffic allowed by default [ 0.759162] vgaarb: loaded [ 0.762126] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.764018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.772000] clocksource: Switched to clocksource kvm-clock [ 0.887141] VFS: Disk quotas dquot_6.6.0 [ 0.889218] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.892152] *** VALIDATE ramfs *** [ 0.893805] *** VALIDATE hugetlbfs *** [ 0.895843] pnp: PnP ACPI init [ 0.898759] pnp: PnP ACPI: found 6 devices [ 0.916885] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.921312] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.924096] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.926798] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.929802] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.932598] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.936247] NET: Registered protocol family 2 [ 0.939087] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.944425] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.948434] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.953704] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.957174] TCP: Hash tables configured (established 65536 bind 65536) [ 0.960448] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.964117] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.967485] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.970394] NET: Registered protocol family 1 [ 0.973284] RPC: Registered named UNIX socket transport module. [ 0.975634] RPC: Registered udp transport module. [ 0.977775] RPC: Registered tcp transport module. [ 0.980142] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.983292] NET: Registered protocol family 44 [ 0.984994] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.986835] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.989060] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.990852] PCI: CLS 0 bytes, default 64 [ 0.993095] Unpacking initramfs... [ 2.449979] debug: unmapping init [mem 0xffff8eb0bcc54000-0xffff8eb0bffbffff] [ 2.454522] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.456602] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.459466] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.961237] Initialise system trusted keyrings [ 2.963686] Key type blacklist registered [ 2.965940] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.975807] zbud: loaded [ 2.979325] *** VALIDATE nfs *** [ 2.980780] *** VALIDATE nfs4 *** [ 2.983080] pstore: using deflate compression [ 2.987320] Platform Keyring initialized [ 3.096265] NET: Registered protocol family 38 [ 3.098523] Key type asymmetric registered [ 3.100447] Asymmetric key parser 'x509' registered [ 3.102870] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.107374] io scheduler mq-deadline registered [ 3.109748] io scheduler kyber registered [ 3.112168] io scheduler bfq registered [ 3.115133] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.118860] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.122426] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.125997] ACPI: Power Button [PWRF] [ 3.132321] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.141267] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.153952] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.159418] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.172316] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.199161] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.229278] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.236362] Non-volatile memory driver v1.3 [ 3.238165] Linux agpgart interface v0.103 [ 3.271349] virtio_blk virtio1: [vda] 150640 512-byte logical blocks (77.1 MB/73.6 MiB) [ 3.274583] vda: detected capacity change from 0 to 77127680 [ 3.291812] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.295680] vdb: detected capacity change from 0 to 1073741824 [ 3.314194] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.318267] vdc: detected capacity change from 0 to 2621440000 [ 3.333855] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.337169] vdd: detected capacity change from 0 to 2621440000 [ 3.351802] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.355618] vde: detected capacity change from 0 to 4294967296 [ 3.369953] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.372928] vdf: detected capacity change from 0 to 4294967296 [ 3.379937] libphy: Fixed MDIO Bus: probed [ 3.387325] usbcore: registered new interface driver usbserial_generic [ 3.390542] usbserial: USB Serial support registered for generic [ 3.393525] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.398602] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.400912] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.403803] mousedev: PS/2 mouse device common for all mice [ 3.407038] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.408671] rtc_cmos 00:05: RTC can wake from S4 [ 3.415191] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.415508] rtc_cmos 00:05: registered as rtc0 [ 3.421205] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.424290] intel_pstate: CPU model not supported [ 3.427047] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.433160] hid: raw HID events driver (C) Jiri Kosina [ 3.435756] usbcore: registered new interface driver usbhid [ 3.437820] usbhid: USB HID core driver [ 3.439064] drop_monitor: Initializing network drop monitor service [ 3.441868] Initializing XFRM netlink socket [ 3.444697] NET: Registered protocol family 10 [ 3.449427] Segment Routing with IPv6 [ 3.451631] NET: Registered protocol family 17 [ 3.454186] mpls_gso: MPLS GSO support [ 3.460622] RAS: Correctable Errors collector initialized. [ 3.463200] AVX version of gcm_enc/dec engaged. [ 3.465318] AES CTR mode by8 optimization enabled [ 3.559062] sched_clock: Marking stable (3559031912, 0)->(4410798796, -851766884) [ 3.563653] registered taskstats version 1 [ 3.565890] Loading compiled-in X.509 certificates [ 3.568238] zswap: loaded using pool lzo/zbud [ 3.594439] Key type big_key registered [ 3.606785] Key type encrypted registered [ 3.608763] ima: No TPM chip found, activating TPM-bypass! [ 3.610895] ima: Allocated hash algorithm: sha1 [ 3.612927] ima: No architecture policies found [ 3.615142] evm: Initialising EVM extended attributes: [ 3.617173] evm: security.selinux [ 3.618191] evm: security.ima [ 3.619447] evm: security.capability [ 3.620855] evm: HMAC attrs: 0x1 [ 3.623710] rtc_cmos 00:05: setting system clock to 2026-09-30 03:51:54 UTC (1790740314) [ 3.630560] debug: unmapping init [mem 0xffffffffb6003000-0xffffffffb61fffff] [ 3.633687] debug: unmapping init [mem 0xffffffffb4d82000-0xffffffffb5058fff] [ 3.643115] Write protecting the kernel read-only data: 28672k [ 3.646989] debug: unmapping init [mem 0xffffffffb3403000-0xffffffffb35fffff] [ 3.650279] debug: unmapping init [mem 0xffffffffb3d14000-0xffffffffb3dfffff] [ 3.690946] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.700512] systemd[1]: Detected virtualization kvm. [ 3.702668] systemd[1]: Detected architecture x86-64. [ 3.704509] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.727899] systemd[1]: No hostname configured. [ 3.729857] systemd[1]: Set hostname to . [ 3.732208] random: systemd: uninitialized urandom read (16 bytes read) [ 3.734547] systemd[1]: Initializing machine ID from random generator. [ 3.789934] random: ln: uninitialized urandom read (6 bytes read) [ 3.871902] random: systemd: uninitialized urandom read (16 bytes read) [ 3.874620] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.880262] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.888300] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.572466] device-mapper: uevent: version 1.0.3 [ 4.575457] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.352281] virtio_net virtio0 ens2: renamed from eth0 [ 5.368391] random: fast init done [ 5.443911] scsi host0: ata_piix [ 5.485286] scsi host1: ata_piix [ 5.490712] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.493731] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.307245] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.126597] random: crng init done [ 10.128373] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.652142] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.851559] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.132468] SELinux: Disabled at runtime. [ 12.197194] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.205431] systemd[1]: Detected virtualization kvm. [ 12.207086] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.779340] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.783522] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.788796] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.794882] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.799137] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.806798] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.811782] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on[ 12.845201] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.288130] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.644132] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.646407] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.778055] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.787681] EDAC sbridge: Ver: 1.1.2 [ 15.447219] Key type dns_resolver registered [ 15.776474] NFS: Registering the id_resolver key type [ 15.779123] Key type id_resolver registered [ 15.781815] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. [ 23.949093] do_IRQ: 0.41 No irq handler for vector [ 25.002152] do_IRQ: 0.41 No irq handler for vector [ 25.012674] do_IRQ: 0.41 No irq handler for vector [ 25.019449] do_IRQ: 0.41 No irq handler for vector [ 25.023847] do_IRQ: 0.41 No irq handler for vector [ 25.026379] do_IRQ: 0.41 No irq handler for vector [ 25.033424] do_IRQ: 0.41 No irq handler for vector [ 25.037340] do_IRQ: 0.41 No irq handler for vector [ 25.039741] do_IRQ: 0.41 No irq handler for vector [ 25.041948] do_IRQ: 0.41 No irq handler for vector [ 67.218446] libcfs: loading out-of-tree module taints kernel. [ 67.226843] do_IRQ: 1 callbacks suppressed [ 67.226852] do_IRQ: 0.41 No irq handler for vector [ 67.238619] do_IRQ: 0.41 No irq handler for vector [ 67.240971] do_IRQ: 0.41 No irq handler for vector [ 67.243483] do_IRQ: 0.41 No irq handler for vector [ 67.245495] do_IRQ: 0.41 No irq handler for vector [ 67.248857] do_IRQ: 0.41 No irq handler for vector [ 67.252584] do_IRQ: 0.41 No irq handler for vector [ 67.260587] do_IRQ: 0.41 No irq handler for vector [ 67.262826] do_IRQ: 0.41 No irq handler for vector [ 67.268832] do_IRQ: 0.41 No irq handler for vector [ 67.288818] Key type ._llcrypt registered [ 67.291957] Key type .llcrypt registered [ 67.393932] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_hostid [ 82.675437] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 82.681588] do_IRQ: 4 callbacks suppressed [ 82.681599] do_IRQ: 0.41 No irq handler for vector [ 82.683198] do_IRQ: 0.41 No irq handler for vector [ 82.684643] do_IRQ: 0.41 No irq handler for vector [ 82.686132] do_IRQ: 0.41 No irq handler for vector [ 82.688073] do_IRQ: 0.41 No irq handler for vector [ 82.690421] do_IRQ: 0.41 No irq handler for vector [ 82.691775] do_IRQ: 0.41 No irq handler for vector [ 82.693770] do_IRQ: 0.41 No irq handler for vector [ 82.695920] do_IRQ: 0.41 No irq handler for vector [ 82.698061] do_IRQ: 0.41 No irq handler for vector [ 83.789396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 83.805333] alg: No test for adler32 (adler32-zlib) [ 85.097166] Lustre: Lustre: Build Version: 2.17.59_3_g21e240e [ 85.753298] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [ 87.520360] Key type lgssc registered [ 88.937110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 88.939571] do_IRQ: 7 callbacks suppressed [ 88.939580] do_IRQ: 0.41 No irq handler for vector [ 88.947889] do_IRQ: 0.41 No irq handler for vector [ 88.951170] do_IRQ: 0.41 No irq handler for vector [ 88.954152] do_IRQ: 0.41 No irq handler for vector [ 88.957108] do_IRQ: 0.41 No irq handler for vector [ 88.959964] do_IRQ: 0.41 No irq handler for vector [ 88.965358] do_IRQ: 0.41 No irq handler for vector [ 88.968738] do_IRQ: 0.41 No irq handler for vector [ 88.971618] do_IRQ: 0.41 No irq handler for vector [ 88.974322] do_IRQ: 0.41 No irq handler for vector [ 105.568487] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 105.577417] do_IRQ: 1 callbacks suppressed [ 105.577427] do_IRQ: 0.41 No irq handler for vector [ 105.584193] do_IRQ: 0.41 No irq handler for vector [ 105.589275] do_IRQ: 0.41 No irq handler for vector [ 105.592979] do_IRQ: 0.41 No irq handler for vector [ 105.597316] do_IRQ: 0.41 No irq handler for vector [ 105.604072] do_IRQ: 0.41 No irq handler for vector [ 105.607292] do_IRQ: 0.41 No irq handler for vector [ 105.610757] do_IRQ: 0.41 No irq handler for vector [ 105.613210] do_IRQ: 0.41 No irq handler for vector [ 105.616530] do_IRQ: 0.41 No irq handler for vector [ 143.476081] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [ 143.480736] do_IRQ: 1 callbacks suppressed [ 143.480747] do_IRQ: 0.41 No irq handler for vector [ 143.483491] do_IRQ: 0.41 No irq handler for vector [ 143.486163] do_IRQ: 0.41 No irq handler for vector [ 143.489019] do_IRQ: 0.41 No irq handler for vector [ 143.492368] do_IRQ: 0.41 No irq handler for vector [ 143.496563] do_IRQ: 0.41 No irq handler for vector [ 143.499747] do_IRQ: 0.41 No irq handler for vector [ 143.502362] do_IRQ: 0.41 No irq handler for vector [ 143.505090] do_IRQ: 0.41 No irq handler for vector [ 143.507916] do_IRQ: 0.41 No irq handler for vector [ 155.633833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 155.654285] do_IRQ: 2 callbacks suppressed [ 155.654294] do_IRQ: 0.41 No irq handler for vector [ 155.658814] do_IRQ: 0.41 No irq handler for vector [ 155.666749] do_IRQ: 0.41 No irq handler for vector [ 155.671174] do_IRQ: 0.41 No irq handler for vector [ 155.673040] do_IRQ: 0.41 No irq handler for vector [ 155.675248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.675853] do_IRQ: 0.41 No irq handler for vector [ 155.683585] do_IRQ: 0.41 No irq handler for vector [ 155.686897] do_IRQ: 0.41 No irq handler for vector [ 155.689387] do_IRQ: 0.41 No irq handler for vector [ 155.692603] do_IRQ: 0.41 No irq handler for vector [ 157.024705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 157.089253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 157.142284] Lustre: lustre-MDT0000: new disk, initializing [ 157.196463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.214557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 160.886455] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.893750] do_IRQ: 6 callbacks suppressed [ 160.893762] do_IRQ: 0.41 No irq handler for vector [ 160.899165] do_IRQ: 0.41 No irq handler for vector [ 160.901677] do_IRQ: 0.41 No irq handler for vector [ 160.904644] do_IRQ: 0.41 No irq handler for vector [ 160.906079] do_IRQ: 0.41 No irq handler for vector [ 160.907554] do_IRQ: 0.41 No irq handler for vector [ 160.909029] do_IRQ: 0.41 No irq handler for vector [ 160.910655] do_IRQ: 0.41 No irq handler for vector [ 160.912350] do_IRQ: 0.41 No irq handler for vector [ 160.913899] do_IRQ: 0.41 No irq handler for vector [ 173.723664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.732136] do_IRQ: 1 callbacks suppressed [ 173.732147] do_IRQ: 0.41 No irq handler for vector [ 173.735105] do_IRQ: 0.41 No irq handler for vector [ 173.741994] do_IRQ: 0.41 No irq handler for vector [ 173.749707] do_IRQ: 0.41 No irq handler for vector [ 173.756729] do_IRQ: 0.41 No irq handler for vector [ 173.759687] do_IRQ: 0.41 No irq handler for vector [ 173.763022] do_IRQ: 0.41 No irq handler for vector [ 173.764838] do_IRQ: 0.41 No irq handler for vector [ 173.767212] do_IRQ: 0.41 No irq handler for vector [ 173.771366] do_IRQ: 0.41 No irq handler for vector [ 173.849618] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 173.873829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 173.878785] Lustre: Skipped 1 previous similar message [ 173.951872] Lustre: lustre-MDT0001: new disk, initializing [ 174.021939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 174.044888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 174.058173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.572878] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.572509] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 181.576810] do_IRQ: 10 callbacks suppressed [ 181.576818] do_IRQ: 0.41 No irq handler for vector [ 181.580918] do_IRQ: 0.41 No irq handler for vector [ 181.582792] do_IRQ: 0.41 No irq handler for vector [ 181.584862] do_IRQ: 0.41 No irq handler for vector [ 181.587021] do_IRQ: 0.41 No irq handler for vector [ 181.589288] do_IRQ: 0.41 No irq handler for vector [ 181.591162] do_IRQ: 0.41 No irq handler for vector [ 181.593832] do_IRQ: 0.41 No irq handler for vector [ 181.595919] do_IRQ: 0.41 No irq handler for vector [ 181.597747] do_IRQ: 0.41 No irq handler for vector [ 189.277581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 189.293684] do_IRQ: 1 callbacks suppressed [ 189.293693] do_IRQ: 0.41 No irq handler for vector [ 189.296487] do_IRQ: 0.41 No irq handler for vector [ 189.298949] do_IRQ: 0.41 No irq handler for vector [ 189.301217] do_IRQ: 0.41 No irq handler for vector [ 189.305372] do_IRQ: 0.41 No irq handler for vector [ 189.316050] do_IRQ: 0.41 No irq handler for vector [ 189.323030] do_IRQ: 0.41 No irq handler for vector [ 189.329424] do_IRQ: 0.41 No irq handler for vector [ 189.347697] do_IRQ: 0.41 No irq handler for vector [ 189.356512] do_IRQ: 0.41 No irq handler for vector [ 189.756720] Lustre: lustre-OST0000: new disk, initializing [ 189.774052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 189.776749] Lustre: 8446:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 189.863301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.996208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.011892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.055802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.203777] hrtimer: interrupt took 5589255 ns [ 195.918867] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.936990] do_IRQ: 10 callbacks suppressed [ 195.936997] do_IRQ: 0.41 No irq handler for vector [ 195.953688] do_IRQ: 0.41 No irq handler for vector [ 195.960795] do_IRQ: 0.41 No irq handler for vector [ 195.970572] do_IRQ: 0.41 No irq handler for vector [ 195.976525] do_IRQ: 0.41 No irq handler for vector [ 195.978782] do_IRQ: 0.41 No irq handler for vector [ 195.985684] do_IRQ: 0.41 No irq handler for vector [ 195.990891] do_IRQ: 0.41 No irq handler for vector [ 195.994754] do_IRQ: 0.41 No irq handler for vector [ 196.000064] do_IRQ: 0.41 No irq handler for vector [ 207.639813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.644833] do_IRQ: 1 callbacks suppressed [ 207.647975] do_IRQ: 0.41 No irq handler for vector [ 207.658336] do_IRQ: 0.41 No irq handler for vector [ 207.663459] do_IRQ: 0.41 No irq handler for vector [ 207.666760] do_IRQ: 0.41 No irq handler for vector [ 207.672116] do_IRQ: 0.41 No irq handler for vector [ 207.675303] do_IRQ: 0.41 No irq handler for vector [ 207.677929] do_IRQ: 0.41 No irq handler for vector [ 207.680429] do_IRQ: 0.41 No irq handler for vector [ 207.682719] do_IRQ: 0.41 No irq handler for vector [ 207.685276] do_IRQ: 0.41 No irq handler for vector [ 207.777811] Lustre: lustre-OST0001: new disk, initializing [ 207.782560] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 207.786630] Lustre: 9517:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.839103] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.522842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 213.537791] do_IRQ: 5 callbacks suppressed [ 213.537802] do_IRQ: 0.41 No irq handler for vector [ 213.540906] do_IRQ: 0.41 No irq handler for vector [ 213.546099] do_IRQ: 0.41 No irq handler for vector [ 213.548925] do_IRQ: 0.41 No irq handler for vector [ 213.555042] do_IRQ: 0.41 No irq handler for vector [ 213.560918] do_IRQ: 0.41 No irq handler for vector [ 213.571076] do_IRQ: 0.41 No irq handler for vector [ 213.580185] do_IRQ: 0.41 No irq handler for vector [ 213.587827] do_IRQ: 0.41 No irq handler for vector [ 213.594056] do_IRQ: 0.41 No irq handler for vector [ 213.604393] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 213.640266] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 213.695547] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.931638] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 225.933706] do_IRQ: 5 callbacks suppressed [ 225.933715] do_IRQ: 0.41 No irq handler for vector [ 225.952986] do_IRQ: 0.41 No irq handler for vector [ 225.957247] do_IRQ: 0.41 No irq handler for vector [ 225.959717] do_IRQ: 0.41 No irq handler for vector [ 225.961305] do_IRQ: 0.41 No irq handler for vector [ 225.964111] do_IRQ: 0.41 No irq handler for vector [ 225.966274] do_IRQ: 0.41 No irq handler for vector [ 225.968482] do_IRQ: 0.41 No irq handler for vector [ 225.971750] do_IRQ: 0.41 No irq handler for vector [ 225.976464] do_IRQ: 0.41 No irq handler for vector [ 234.261968] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 234.264650] do_IRQ: 1 callbacks suppressed [ 234.264659] do_IRQ: 0.41 No irq handler for vector [ 234.267606] do_IRQ: 0.41 No irq handler for vector [ 234.276876] do_IRQ: 0.41 No irq handler for vector [ 234.282137] do_IRQ: 0.41 No irq handler for vector [ 234.288250] do_IRQ: 0.41 No irq handler for vector [ 234.291185] do_IRQ: 0.41 No irq handler for vector [ 234.295164] do_IRQ: 0.41 No irq handler for vector [ 234.298290] do_IRQ: 0.41 No irq handler for vector [ 234.301575] do_IRQ: 0.41 No irq handler for vector [ 234.303801] do_IRQ: 0.41 No irq handler for vector [ 240.135817] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing check_logdir /tmp/testlogs/ [ 240.149078] do_IRQ: 1 callbacks suppressed [ 240.149086] do_IRQ: 0.41 No irq handler for vector [ 240.157510] do_IRQ: 0.41 No irq handler for vector [ 240.163942] do_IRQ: 0.41 No irq handler for vector [ 240.169235] do_IRQ: 0.41 No irq handler for vector [ 240.190285] do_IRQ: 0.41 No irq handler for vector [ 240.192228] do_IRQ: 0.41 No irq handler for vector [ 240.201367] do_IRQ: 0.41 No irq handler for vector [ 240.210323] do_IRQ: 0.41 No irq handler for vector [ 240.215145] do_IRQ: 0.41 No irq handler for vector [ 240.221096] do_IRQ: 0.41 No irq handler for vector [ 245.327333] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing yml_node [ 245.337432] do_IRQ: 1 callbacks suppressed [ 245.337443] do_IRQ: 0.41 No irq handler for vector [ 245.348088] do_IRQ: 0.41 No irq handler for vector [ 245.353190] do_IRQ: 0.41 No irq handler for vector [ 245.355494] do_IRQ: 0.41 No irq handler for vector [ 245.359251] do_IRQ: 0.41 No irq handler for vector [ 245.362772] do_IRQ: 0.41 No irq handler for vector [ 245.372604] do_IRQ: 0.41 No irq handler for vector [ 245.380565] do_IRQ: 0.41 No irq handler for vector [ 245.388068] do_IRQ: 0.41 No irq handler for vector [ 245.394909] do_IRQ: 0.41 No irq handler for vector [ 250.027203] Lustre: DEBUG MARKER: Client: 2.17.59.3 [ 252.644907] Lustre: DEBUG MARKER: MDS: 2.17.59.3 [ 252.657215] do_IRQ: 3 callbacks suppressed [ 252.657225] do_IRQ: 0.41 No irq handler for vector [ 252.661088] do_IRQ: 0.41 No irq handler for vector [ 252.667781] do_IRQ: 0.41 No irq handler for vector [ 252.670884] do_IRQ: 0.41 No irq handler for vector [ 252.673273] do_IRQ: 0.41 No irq handler for vector [ 252.675904] do_IRQ: 0.41 No irq handler for vector [ 252.682828] do_IRQ: 0.41 No irq handler for vector [ 252.693250] do_IRQ: 0.41 No irq handler for vector [ 252.701623] do_IRQ: 0.41 No irq handler for vector [ 252.707274] do_IRQ: 0.41 No irq handler for vector [ 255.324140] Lustre: DEBUG MARKER: OSS: 2.17.59.3 [ 256.859657] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Tue Sep 29 23:56:05 EDT 2026 [ 272.709059] Lustre: DEBUG MARKER: - need MDS1_VERSION <= 2.16.61-1-g89cf292a8c2 (34683651 <= 34618625) for LU-18938, skip 360 [ 272.714330] do_IRQ: 4 callbacks suppressed [ 272.714341] do_IRQ: 0.41 No irq handler for vector [ 272.725457] do_IRQ: 0.41 No irq handler for vector [ 272.731273] do_IRQ: 0.41 No irq handler for vector [ 272.737403] do_IRQ: 0.41 No irq handler for vector [ 272.740715] do_IRQ: 0.41 No irq handler for vector [ 272.748981] do_IRQ: 0.41 No irq handler for vector [ 272.752144] do_IRQ: 0.41 No irq handler for vector [ 272.755353] do_IRQ: 0.41 No irq handler for vector [ 272.764652] do_IRQ: 0.41 No irq handler for vector [ 272.778642] do_IRQ: 0.41 No irq handler for vector [ 274.188890] Lustre: DEBUG MARKER: - need MDS1_VERSION < v2_14_55-100-g8a84c7f9c7 (34683651 < 34486116) for LU-14927, skip 0f [ 275.629591] Lustre: DEBUG MARKER: - need OST1_VERSION < v2_17_50-410-gbd8e7b9555 (34683651 < 34681754) for LU-12550, skip 216 [ 276.777365] Lustre: DEBUG MARKER: only running tests: 60-199 [ 278.020299] Lustre: DEBUG MARKER: excepting tests: 42a 42c 42b 118c 118d 407 119i 216 817 411a 360 0f 216 [ 278.025835] do_IRQ: 5 callbacks suppressed [ 278.025846] do_IRQ: 0.41 No irq handler for vector [ 278.036623] do_IRQ: 0.41 No irq handler for vector [ 278.044038] do_IRQ: 0.41 No irq handler for vector [ 278.049751] do_IRQ: 0.41 No irq handler for vector [ 278.052994] do_IRQ: 0.41 No irq handler for vector [ 278.056940] do_IRQ: 0.41 No irq handler for vector [ 278.065407] do_IRQ: 0.41 No irq handler for vector [ 278.068537] do_IRQ: 0.41 No irq handler for vector [ 278.073269] do_IRQ: 0.41 No irq handler for vector [ 278.078200] do_IRQ: 0.41 No irq handler for vector [ 279.357921] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 51c 51e 834 [ 280.681514] Lustre: DEBUG MARKER: === sanity: start setup 23:56:30 (1790740590) === [ 285.277853] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing check_config_client /mnt/lustre [ 285.282940] do_IRQ: 3 callbacks suppressed [ 285.282954] do_IRQ: 0.41 No irq handler for vector [ 285.292390] do_IRQ: 0.41 No irq handler for vector [ 285.298059] do_IRQ: 0.41 No irq handler for vector [ 285.303900] do_IRQ: 0.41 No irq handler for vector [ 285.308360] do_IRQ: 0.41 No irq handler for vector [ 285.314111] do_IRQ: 0.41 No irq handler for vector [ 285.320341] do_IRQ: 0.41 No irq handler for vector [ 285.322280] do_IRQ: 0.41 No irq handler for vector [ 285.324680] do_IRQ: 0.41 No irq handler for vector [ 285.328320] do_IRQ: 0.41 No irq handler for vector [ 300.196842] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 300.199481] do_IRQ: 1 callbacks suppressed [ 300.199491] do_IRQ: 0.41 No irq handler for vector [ 300.210286] do_IRQ: 0.41 No irq handler for vector [ 300.213984] do_IRQ: 0.41 No irq handler for vector [ 300.220024] do_IRQ: 0.41 No irq handler for vector [ 300.222411] do_IRQ: 0.41 No irq handler for vector [ 300.225356] do_IRQ: 0.41 No irq handler for vector [ 300.227115] do_IRQ: 0.41 No irq handler for vector [ 300.231127] do_IRQ: 0.41 No irq handler for vector [ 300.233483] do_IRQ: 0.41 No irq handler for vector [ 300.237962] do_IRQ: 0.41 No irq handler for vector [ 302.736840] Lustre: 13567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 307.309970] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 307.334511] do_IRQ: 2 callbacks suppressed [ 307.334523] do_IRQ: 0.41 No irq handler for vector [ 307.342507] do_IRQ: 0.41 No irq handler for vector [ 307.349501] do_IRQ: 0.41 No irq handler for vector [ 307.353637] do_IRQ: 0.41 No irq handler for vector [ 307.358352] do_IRQ: 0.41 No irq handler for vector [ 307.361610] do_IRQ: 0.41 No irq handler for vector [ 307.367089] do_IRQ: 0.41 No irq handler for vector [ 307.371581] do_IRQ: 0.41 No irq handler for vector [ 307.375192] do_IRQ: 0.41 No irq handler for vector [ 307.378634] do_IRQ: 0.41 No irq handler for vector [ 312.885542] Lustre: DEBUG MARKER: === sanity: finish setup 23:57:01 (1790740621) === [ 312.891125] do_IRQ: 1 callbacks suppressed [ 312.891134] do_IRQ: 0.41 No irq handler for vector [ 312.904800] do_IRQ: 0.41 No irq handler for vector [ 312.912206] do_IRQ: 0.41 No irq handler for vector [ 312.918162] do_IRQ: 0.41 No irq handler for vector [ 312.924049] do_IRQ: 0.41 No irq handler for vector [ 312.926417] do_IRQ: 0.41 No irq handler for vector [ 312.929366] do_IRQ: 0.41 No irq handler for vector [ 312.935111] do_IRQ: 0.41 No irq handler for vector [ 312.939827] do_IRQ: 0.41 No irq handler for vector [ 312.942334] do_IRQ: 0.41 No irq handler for vector [ 319.946663] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 23:57:08 (1790740628) [ 319.959042] do_IRQ: 1 callbacks suppressed [ 319.959051] do_IRQ: 0.41 No irq handler for vector [ 319.961897] do_IRQ: 0.41 No irq handler for vector [ 319.970654] do_IRQ: 0.41 No irq handler for vector [ 319.973233] do_IRQ: 0.41 No irq handler for vector [ 319.976850] do_IRQ: 0.41 No irq handler for vector [ 319.979785] do_IRQ: 0.41 No irq handler for vector [ 319.981899] do_IRQ: 0.41 No irq handler for vector [ 319.985552] do_IRQ: 0.41 No irq handler for vector [ 319.993734] do_IRQ: 0.41 No irq handler for vector [ 320.000542] do_IRQ: 0.41 No irq handler for vector [ 322.978651] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 324.442230] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 23:57:13 (1790740633) [ 331.655325] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 23:57:20 (1790740640) [ 331.662936] do_IRQ: 3 callbacks suppressed [ 331.662956] do_IRQ: 0.41 No irq handler for vector [ 331.670013] do_IRQ: 0.41 No irq handler for vector [ 331.673459] do_IRQ: 0.41 No irq handler for vector [ 331.681247] do_IRQ: 0.41 No irq handler for vector [ 331.683477] do_IRQ: 0.41 No irq handler for vector [ 331.686750] do_IRQ: 0.41 No irq handler for vector [ 331.689167] do_IRQ: 0.41 No irq handler for vector [ 331.691687] do_IRQ: 0.41 No irq handler for vector [ 331.694368] do_IRQ: 0.41 No irq handler for vector [ 331.697019] do_IRQ: 0.41 No irq handler for vector [ 474.826433] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 23:59:44 (1790740784) [ 474.838710] do_IRQ: 1 callbacks suppressed [ 474.838719] do_IRQ: 0.41 No irq handler for vector [ 474.841839] do_IRQ: 0.41 No irq handler for vector [ 474.848377] do_IRQ: 0.41 No irq handler for vector [ 474.852757] do_IRQ: 0.41 No irq handler for vector [ 474.857621] do_IRQ: 0.41 No irq handler for vector [ 474.863003] do_IRQ: 0.41 No irq handler for vector [ 474.867523] do_IRQ: 0.41 No irq handler for vector [ 474.870980] do_IRQ: 0.41 No irq handler for vector [ 474.873872] do_IRQ: 0.41 No irq handler for vector [ 474.875983] do_IRQ: 0.41 No irq handler for vector [ 483.304953] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 23:59:52 (1790740792) [ 483.332948] do_IRQ: 1 callbacks suppressed [ 483.332959] do_IRQ: 0.41 No irq handler for vector [ 483.347676] do_IRQ: 0.41 No irq handler for vector [ 483.360304] do_IRQ: 0.41 No irq handler for vector [ 483.364865] do_IRQ: 0.41 No irq handler for vector [ 483.368952] do_IRQ: 0.41 No irq handler for vector [ 483.381824] do_IRQ: 0.41 No irq handler for vector [ 483.403361] do_IRQ: 0.41 No irq handler for vector [ 483.412354] do_IRQ: 0.41 No irq handler for vector [ 483.420506] do_IRQ: 0.41 No irq handler for vector [ 483.429637] do_IRQ: 0.41 No irq handler for vector [ 485.281972] Lustre: *** cfs_fail_loc=15b, val=0*** [ 485.285656] Lustre: *** cfs_fail_loc=15b, val=0*** [ 485.289704] LustreError: 10303:0:(llog_cat_server.c:483:llog_cat_add_rec()) lustre-OST0001-osc-MDT0000: initialization error: rc = -28 [ 493.640282] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 00:00:02 (1790740802) [ 493.664116] do_IRQ: 4 callbacks suppressed [ 493.664125] do_IRQ: 0.41 No irq handler for vector [ 493.672732] do_IRQ: 0.41 No irq handler for vector [ 493.681612] do_IRQ: 0.41 No irq handler for vector [ 493.698426] do_IRQ: 0.41 No irq handler for vector [ 493.710397] do_IRQ: 0.41 No irq handler for vector [ 493.723353] do_IRQ: 0.41 No irq handler for vector [ 493.733285] do_IRQ: 0.41 No irq handler for vector [ 493.740668] do_IRQ: 0.41 No irq handler for vector [ 493.742900] do_IRQ: 0.41 No irq handler for vector [ 493.748737] do_IRQ: 0.41 No irq handler for vector [ 501.769352] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 00:00:11 (1790740811) [ 501.779329] do_IRQ: 2 callbacks suppressed [ 501.779340] do_IRQ: 0.41 No irq handler for vector [ 501.789123] do_IRQ: 0.41 No irq handler for vector [ 501.793975] do_IRQ: 0.41 No irq handler for vector [ 501.798916] do_IRQ: 0.41 No irq handler for vector [ 501.802216] do_IRQ: 0.41 No irq handler for vector [ 501.804984] do_IRQ: 0.41 No irq handler for vector [ 501.808139] do_IRQ: 0.41 No irq handler for vector [ 501.816514] do_IRQ: 0.41 No irq handler for vector [ 501.821867] do_IRQ: 0.41 No irq handler for vector [ 501.825996] do_IRQ: 0.41 No irq handler for vector [ 502.802106] Lustre: *** cfs_fail_loc=19a, val=0*** [ 503.641718] Lustre: *** cfs_fail_loc=19a, val=0*** [ 505.326836] Lustre: *** cfs_fail_loc=19a, val=0*** [ 505.331750] Lustre: Skipped 1 previous similar message [ 507.672357] Lustre: *** cfs_fail_loc=19a, val=0*** [ 507.674272] Lustre: Skipped 2 previous similar messages [ 507.676883] do_IRQ: 5 callbacks suppressed [ 507.676892] do_IRQ: 0.41 No irq handler for vector [ 507.701370] do_IRQ: 0.41 No irq handler for vector [ 507.705328] do_IRQ: 0.41 No irq handler for vector [ 507.715096] do_IRQ: 0.41 No irq handler for vector [ 507.716967] do_IRQ: 0.41 No irq handler for vector [ 507.723491] do_IRQ: 0.41 No irq handler for vector [ 507.727872] do_IRQ: 0.41 No irq handler for vector [ 507.730492] do_IRQ: 0.41 No irq handler for vector [ 507.734571] do_IRQ: 0.41 No irq handler for vector [ 507.736218] do_IRQ: 0.41 No irq handler for vector [ 512.317529] Lustre: *** cfs_fail_loc=19a, val=0*** [ 512.324595] Lustre: Skipped 4 previous similar messages [ 520.691670] Lustre: *** cfs_fail_loc=19a, val=0*** [ 520.701314] Lustre: Skipped 7 previous similar messages [ 520.703184] do_IRQ: 3 callbacks suppressed [ 520.703192] do_IRQ: 0.41 No irq handler for vector [ 520.715200] do_IRQ: 0.41 No irq handler for vector [ 520.726173] do_IRQ: 0.41 No irq handler for vector [ 520.736518] do_IRQ: 0.41 No irq handler for vector [ 520.747111] do_IRQ: 0.41 No irq handler for vector [ 520.752080] do_IRQ: 0.41 No irq handler for vector [ 520.758955] do_IRQ: 0.41 No irq handler for vector [ 520.768637] do_IRQ: 0.41 No irq handler for vector [ 520.770437] do_IRQ: 0.41 No irq handler for vector [ 520.784225] do_IRQ: 0.41 No irq handler for vector [ 537.202368] Lustre: *** cfs_fail_loc=19a, val=0*** [ 537.205602] Lustre: Skipped 16 previous similar messages [ 537.207462] do_IRQ: 1 callbacks suppressed [ 537.207478] do_IRQ: 0.41 No irq handler for vector [ 537.214462] do_IRQ: 0.41 No irq handler for vector [ 537.219577] do_IRQ: 0.41 No irq handler for vector [ 537.226072] do_IRQ: 0.41 No irq handler for vector [ 537.232611] do_IRQ: 0.41 No irq handler for vector [ 537.235369] do_IRQ: 0.41 No irq handler for vector [ 537.238851] do_IRQ: 0.41 No irq handler for vector [ 537.242008] do_IRQ: 0.41 No irq handler for vector [ 537.250745] do_IRQ: 0.41 No irq handler for vector [ 537.257736] do_IRQ: 0.41 No irq handler for vector [ 569.525047] Lustre: *** cfs_fail_loc=19a, val=0*** [ 569.526943] Lustre: Skipped 34 previous similar messages [ 569.529569] do_IRQ: 1 callbacks suppressed [ 569.529579] do_IRQ: 0.41 No irq handler for vector [ 569.559940] do_IRQ: 0.41 No irq handler for vector [ 569.570469] do_IRQ: 0.41 No irq handler for vector [ 569.576697] do_IRQ: 0.41 No irq handler for vector [ 569.586858] do_IRQ: 0.41 No irq handler for vector [ 569.597489] do_IRQ: 0.41 No irq handler for vector [ 569.604847] do_IRQ: 0.41 No irq handler for vector [ 569.620523] do_IRQ: 0.41 No irq handler for vector [ 569.636173] do_IRQ: 0.41 No irq handler for vector [ 569.647515] do_IRQ: 0.41 No irq handler for vector [ 606.586456] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 00:01:55 (1790740915) [ 606.613043] do_IRQ: 1 callbacks suppressed [ 606.613053] do_IRQ: 0.41 No irq handler for vector [ 606.618201] do_IRQ: 0.41 No irq handler for vector [ 606.631515] do_IRQ: 0.41 No irq handler for vector [ 606.635129] do_IRQ: 0.41 No irq handler for vector [ 606.644444] do_IRQ: 0.41 No irq handler for vector [ 606.653498] do_IRQ: 0.41 No irq handler for vector [ 606.659479] do_IRQ: 0.41 No irq handler for vector [ 606.661732] do_IRQ: 0.41 No irq handler for vector [ 606.668868] do_IRQ: 0.41 No irq handler for vector [ 606.678502] do_IRQ: 0.41 No irq handler for vector [ 607.853738] Lustre: *** cfs_fail_loc=188, val=0*** [ 611.121474] Lustre: *** cfs_fail_loc=189, val=0*** [ 621.646066] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 621.651250] do_IRQ: 3 callbacks suppressed [ 621.651260] do_IRQ: 0.41 No irq handler for vector [ 621.661324] do_IRQ: 0.41 No irq handler for vector [ 621.666775] do_IRQ: 0.41 No irq handler for vector [ 621.673462] do_IRQ: 0.41 No irq handler for vector [ 621.681973] do_IRQ: 0.41 No irq handler for vector [ 621.685760] do_IRQ: 0.41 No irq handler for vector [ 621.692547] do_IRQ: 0.41 No irq handler for vector [ 621.694870] do_IRQ: 0.41 No irq handler for vector [ 621.697017] do_IRQ: 0.41 No irq handler for vector [ 621.702574] do_IRQ: 0.41 No irq handler for vector [ 623.270939] Lustre: DEBUG MARKER: == sanity test 60j: llog_reader reports corruptions ====== 00:02:12 (1790740932) [ 628.547985] Lustre: lustre-MDD0000: changelog on [ 628.554228] do_IRQ: 3 callbacks suppressed [ 628.554238] do_IRQ: 0.41 No irq handler for vector [ 628.556451] do_IRQ: 0.41 No irq handler for vector [ 628.560517] do_IRQ: 0.41 No irq handler for vector [ 628.563228] do_IRQ: 0.41 No irq handler for vector [ 628.565972] do_IRQ: 0.41 No irq handler for vector [ 628.569637] do_IRQ: 0.41 No irq handler for vector [ 628.572506] do_IRQ: 0.41 No irq handler for vector [ 628.575360] do_IRQ: 0.41 No irq handler for vector [ 628.578675] do_IRQ: 0.41 No irq handler for vector [ 628.581985] do_IRQ: 0.41 No irq handler for vector [ 630.988748] Lustre: lustre-MDD0001: changelog on [ 638.495778] Lustre: DEBUG MARKER: SKIP: sanity test_60j path oi.1/0x1:0xc:0x0 is not in 'O/1/d/' format [ 638.515621] do_IRQ: 3 callbacks suppressed [ 638.515629] do_IRQ: 0.41 No irq handler for vector [ 638.523391] do_IRQ: 0.41 No irq handler for vector [ 638.527559] do_IRQ: 0.41 No irq handler for vector [ 638.532582] do_IRQ: 0.41 No irq handler for vector [ 638.536555] do_IRQ: 0.41 No irq handler for vector [ 638.541884] do_IRQ: 0.41 No irq handler for vector [ 638.548339] do_IRQ: 0.41 No irq handler for vector [ 638.550564] do_IRQ: 0.41 No irq handler for vector [ 638.555562] do_IRQ: 0.41 No irq handler for vector [ 638.560430] do_IRQ: 0.41 No irq handler for vector [ 641.410762] Lustre: lustre-MDD0001: changelog off [ 644.515556] Lustre: lustre-MDD0000: changelog off [ 644.518255] do_IRQ: 2 callbacks suppressed [ 644.518265] do_IRQ: 0.41 No irq handler for vector [ 644.537027] do_IRQ: 0.41 No irq handler for vector [ 644.539562] do_IRQ: 0.41 No irq handler for vector [ 644.542675] do_IRQ: 0.41 No irq handler for vector [ 644.547879] do_IRQ: 0.41 No irq handler for vector [ 644.550523] do_IRQ: 0.41 No irq handler for vector [ 644.553557] do_IRQ: 0.41 No irq handler for vector [ 644.562082] do_IRQ: 0.41 No irq handler for vector [ 644.564855] do_IRQ: 0.41 No irq handler for vector [ 644.569761] do_IRQ: 0.41 No irq handler for vector [ 647.187151] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 00:02:36 (1790740956) [ 653.220980] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 00:02:42 (1790740962) [ 653.241900] do_IRQ: 2 callbacks suppressed [ 653.241909] do_IRQ: 0.41 No irq handler for vector [ 653.246596] do_IRQ: 0.41 No irq handler for vector [ 653.249781] do_IRQ: 0.41 No irq handler for vector [ 653.260325] do_IRQ: 0.41 No irq handler for vector [ 653.270378] do_IRQ: 0.41 No irq handler for vector [ 653.281651] do_IRQ: 0.41 No irq handler for vector [ 653.287728] do_IRQ: 0.41 No irq handler for vector [ 653.293666] do_IRQ: 0.41 No irq handler for vector [ 653.297678] do_IRQ: 0.41 No irq handler for vector [ 653.302471] do_IRQ: 0.41 No irq handler for vector [ 659.567894] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 00:02:48 (1790740968) [ 659.587210] do_IRQ: 1 callbacks suppressed [ 659.587219] do_IRQ: 0.41 No irq handler for vector [ 659.593880] do_IRQ: 0.41 No irq handler for vector [ 659.596359] do_IRQ: 0.41 No irq handler for vector [ 659.601190] do_IRQ: 0.41 No irq handler for vector [ 659.603154] do_IRQ: 0.41 No irq handler for vector [ 659.606604] do_IRQ: 0.41 No irq handler for vector [ 659.610638] do_IRQ: 0.41 No irq handler for vector [ 659.612807] do_IRQ: 0.41 No irq handler for vector [ 659.615828] do_IRQ: 0.41 No irq handler for vector [ 659.619374] do_IRQ: 0.41 No irq handler for vector [ 728.471626] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 00:03:57 (1790741037) [ 728.482578] do_IRQ: 1 callbacks suppressed [ 728.482588] do_IRQ: 0.41 No irq handler for vector [ 728.483938] do_IRQ: 0.41 No irq handler for vector [ 728.489684] do_IRQ: 0.41 No irq handler for vector [ 728.492863] do_IRQ: 0.41 No irq handler for vector [ 728.495725] do_IRQ: 0.41 No irq handler for vector [ 728.498766] do_IRQ: 0.41 No irq handler for vector [ 728.501752] do_IRQ: 0.41 No irq handler for vector [ 728.505808] do_IRQ: 0.41 No irq handler for vector [ 728.509765] do_IRQ: 0.41 No irq handler for vector [ 728.512873] do_IRQ: 0.41 No irq handler for vector [ 740.555650] Lustre: DEBUG MARKER: == sanity test 63c: test sync_on_close=1 ================= 00:04:09 (1790741049) [ 740.564157] do_IRQ: 2 callbacks suppressed [ 740.564166] do_IRQ: 0.41 No irq handler for vector [ 740.569677] do_IRQ: 0.41 No irq handler for vector [ 740.576791] do_IRQ: 0.41 No irq handler for vector [ 740.579673] do_IRQ: 0.41 No irq handler for vector [ 740.580833] do_IRQ: 0.41 No irq handler for vector [ 740.583890] do_IRQ: 0.41 No irq handler for vector [ 740.585813] do_IRQ: 0.41 No irq handler for vector [ 740.590839] do_IRQ: 0.41 No irq handler for vector [ 740.593897] do_IRQ: 0.41 No irq handler for vector [ 740.596833] do_IRQ: 0.41 No irq handler for vector [ 754.950967] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 00:04:24 (1790741064) [ 754.959916] do_IRQ: 1 callbacks suppressed [ 754.959928] do_IRQ: 0.41 No irq handler for vector [ 754.969257] do_IRQ: 0.41 No irq handler for vector [ 754.972460] do_IRQ: 0.41 No irq handler for vector [ 754.975598] do_IRQ: 0.41 No irq handler for vector [ 754.982646] do_IRQ: 0.41 No irq handler for vector [ 754.985315] do_IRQ: 0.41 No irq handler for vector [ 754.987996] do_IRQ: 0.41 No irq handler for vector [ 754.992720] do_IRQ: 0.41 No irq handler for vector [ 754.994227] do_IRQ: 0.41 No irq handler for vector [ 754.997137] do_IRQ: 0.41 No irq handler for vector [ 761.792222] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 761.799432] do_IRQ: 1 callbacks suppressed [ 761.799442] do_IRQ: 0.41 No irq handler for vector [ 761.803488] do_IRQ: 0.41 No irq handler for vector [ 761.809985] do_IRQ: 0.41 No irq handler for vector [ 761.819707] do_IRQ: 0.41 No irq handler for vector [ 761.823392] do_IRQ: 0.41 No irq handler for vector [ 761.827284] do_IRQ: 0.41 No irq handler for vector [ 761.830259] do_IRQ: 0.41 No irq handler for vector [ 761.833761] do_IRQ: 0.41 No irq handler for vector [ 761.836581] do_IRQ: 0.41 No irq handler for vector [ 761.843940] do_IRQ: 0.41 No irq handler for vector [ 763.381196] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 00:04:32 (1790741072) [ 770.495212] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 00:04:39 (1790741079) [ 770.500983] do_IRQ: 3 callbacks suppressed [ 770.500993] do_IRQ: 0.41 No irq handler for vector [ 770.514432] do_IRQ: 0.41 No irq handler for vector [ 770.517514] do_IRQ: 0.41 No irq handler for vector [ 770.519241] do_IRQ: 0.41 No irq handler for vector [ 770.525151] do_IRQ: 0.41 No irq handler for vector [ 770.528545] do_IRQ: 0.41 No irq handler for vector [ 770.533232] do_IRQ: 0.41 No irq handler for vector [ 770.536664] do_IRQ: 0.41 No irq handler for vector [ 770.540932] do_IRQ: 0.41 No irq handler for vector [ 770.545423] do_IRQ: 0.41 No irq handler for vector [ 822.215806] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 00:05:30 (1790741130) [ 822.224337] do_IRQ: 1 callbacks suppressed [ 822.224349] do_IRQ: 0.41 No irq handler for vector [ 822.232139] do_IRQ: 0.41 No irq handler for vector [ 822.235062] do_IRQ: 0.41 No irq handler for vector [ 822.237583] do_IRQ: 0.41 No irq handler for vector [ 822.245030] do_IRQ: 0.41 No irq handler for vector [ 822.248333] do_IRQ: 0.41 No irq handler for vector [ 822.251651] do_IRQ: 0.41 No irq handler for vector [ 822.259484] do_IRQ: 0.41 No irq handler for vector [ 822.262188] do_IRQ: 0.41 No irq handler for vector [ 822.265104] do_IRQ: 0.41 No irq handler for vector [ 828.797937] Lustre: *** cfs_fail_loc=725, val=0*** [ 828.809890] do_IRQ: 1 callbacks suppressed [ 828.809900] do_IRQ: 0.41 No irq handler for vector [ 828.838152] do_IRQ: 0.41 No irq handler for vector [ 828.842692] do_IRQ: 0.41 No irq handler for vector [ 828.845923] do_IRQ: 0.41 No irq handler for vector [ 828.853647] do_IRQ: 0.41 No irq handler for vector [ 828.864823] do_IRQ: 0.41 No irq handler for vector [ 828.880385] do_IRQ: 0.41 No irq handler for vector [ 828.889845] do_IRQ: 0.41 No irq handler for vector [ 828.901908] do_IRQ: 0.41 No irq handler for vector [ 828.907839] do_IRQ: 0.41 No irq handler for vector [ 837.141121] Lustre: *** cfs_fail_loc=725, val=0*** [ 837.145375] do_IRQ: 1 callbacks suppressed [ 837.145386] do_IRQ: 0.41 No irq handler for vector [ 837.154414] do_IRQ: 0.41 No irq handler for vector [ 837.157595] do_IRQ: 0.41 No irq handler for vector [ 837.159746] do_IRQ: 0.41 No irq handler for vector [ 837.171674] do_IRQ: 0.41 No irq handler for vector [ 837.177801] do_IRQ: 0.41 No irq handler for vector [ 837.187265] do_IRQ: 0.41 No irq handler for vector [ 837.188819] do_IRQ: 0.41 No irq handler for vector [ 837.195652] do_IRQ: 0.41 No irq handler for vector [ 837.210885] do_IRQ: 0.41 No irq handler for vector [ 844.171882] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 00:05:53 (1790741153) [ 844.197813] do_IRQ: 1 callbacks suppressed [ 844.197822] do_IRQ: 0.41 No irq handler for vector [ 844.210070] do_IRQ: 0.41 No irq handler for vector [ 844.217603] do_IRQ: 0.41 No irq handler for vector [ 844.226880] do_IRQ: 0.41 No irq handler for vector [ 844.234507] do_IRQ: 0.41 No irq handler for vector [ 844.242647] do_IRQ: 0.41 No irq handler for vector [ 844.249987] do_IRQ: 0.41 No irq handler for vector [ 844.254790] do_IRQ: 0.41 No irq handler for vector [ 844.258529] do_IRQ: 0.41 No irq handler for vector [ 844.260809] do_IRQ: 0.41 No irq handler for vector [ 855.553397] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 00:06:04 (1790741164) [ 855.558878] do_IRQ: 1 callbacks suppressed [ 855.558889] do_IRQ: 0.41 No irq handler for vector [ 855.571033] do_IRQ: 0.41 No irq handler for vector [ 855.580241] do_IRQ: 0.41 No irq handler for vector [ 855.583123] do_IRQ: 0.41 No irq handler for vector [ 855.585359] do_IRQ: 0.41 No irq handler for vector [ 855.598433] do_IRQ: 0.41 No irq handler for vector [ 855.601659] do_IRQ: 0.41 No irq handler for vector [ 855.606711] do_IRQ: 0.41 No irq handler for vector [ 855.608368] do_IRQ: 0.41 No irq handler for vector [ 855.627129] do_IRQ: 0.41 No irq handler for vector [ 903.897808] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 00:06:53 (1790741213) [ 903.903202] do_IRQ: 1 callbacks suppressed [ 903.903212] do_IRQ: 0.41 No irq handler for vector [ 903.912329] do_IRQ: 0.41 No irq handler for vector [ 903.916794] do_IRQ: 0.41 No irq handler for vector [ 903.921534] do_IRQ: 0.41 No irq handler for vector [ 903.923030] do_IRQ: 0.41 No irq handler for vector [ 903.924503] do_IRQ: 0.41 No irq handler for vector [ 903.926210] do_IRQ: 0.41 No irq handler for vector [ 903.927366] do_IRQ: 0.41 No irq handler for vector [ 903.928351] do_IRQ: 0.41 No irq handler for vector [ 903.929657] do_IRQ: 0.41 No irq handler for vector [ 924.715612] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 00:07:13 (1790741233) [ 924.730788] do_IRQ: 1 callbacks suppressed [ 924.730795] do_IRQ: 0.41 No irq handler for vector [ 924.743770] do_IRQ: 0.41 No irq handler for vector [ 924.747046] do_IRQ: 0.41 No irq handler for vector [ 924.749837] do_IRQ: 0.41 No irq handler for vector [ 924.754760] do_IRQ: 0.41 No irq handler for vector [ 924.760322] do_IRQ: 0.41 No irq handler for vector [ 924.766543] do_IRQ: 0.41 No irq handler for vector [ 924.771360] do_IRQ: 0.41 No irq handler for vector [ 924.777052] do_IRQ: 0.41 No irq handler for vector [ 924.781682] do_IRQ: 0.41 No irq handler for vector [ 933.837423] Lustre: *** cfs_fail_loc=513, val=17*** [ 933.843571] do_IRQ: 1 callbacks suppressed [ 933.843582] do_IRQ: 0.41 No irq handler for vector [ 933.846031] do_IRQ: 0.41 No irq handler for vector [ 933.857672] do_IRQ: 0.41 No irq handler for vector [ 933.864837] do_IRQ: 0.41 No irq handler for vector [ 933.875963] do_IRQ: 0.41 No irq handler for vector [ 933.882433] do_IRQ: 0.41 No irq handler for vector [ 933.884479] do_IRQ: 0.41 No irq handler for vector [ 933.887607] do_IRQ: 0.41 No irq handler for vector [ 933.891855] do_IRQ: 0.41 No irq handler for vector [ 933.896274] do_IRQ: 0.41 No irq handler for vector [ 933.907961] LustreError: 14696:0:(service.c:2364:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1877727392122880 [ 935.751276] Lustre: Failing over lustre-OST0000 [ 935.996308] Lustre: server umount lustre-OST0000 complete [ 939.488677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 939.489900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 939.502111] LustreError: Skipped 1 previous similar message [ 939.502121] do_IRQ: 5 callbacks suppressed [ 939.502125] do_IRQ: 0.41 No irq handler for vector [ 939.519648] do_IRQ: 0.41 No irq handler for vector [ 939.521086] Lustre: Skipped 1 previous similar message [ 939.539042] do_IRQ: 0.41 No irq handler for vector [ 939.549175] do_IRQ: 0.41 No irq handler for vector [ 939.553255] do_IRQ: 0.41 No irq handler for vector [ 939.556976] do_IRQ: 0.41 No irq handler for vector [ 939.560870] do_IRQ: 0.41 No irq handler for vector [ 939.564028] do_IRQ: 0.41 No irq handler for vector [ 939.566770] do_IRQ: 0.41 No irq handler for vector [ 939.569982] do_IRQ: 0.41 No irq handler for vector [ 941.250502] LustreError: 8430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 941.277687] LustreError: 8430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 945.637513] LustreError: 14706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 945.660428] do_IRQ: 3 callbacks suppressed [ 945.660427] LustreError: 14706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 945.660437] do_IRQ: 0.41 No irq handler for vector [ 945.661686] do_IRQ: 0.41 No irq handler for vector [ 945.684514] do_IRQ: 0.41 No irq handler for vector [ 945.687092] do_IRQ: 0.41 No irq handler for vector [ 945.689801] do_IRQ: 0.41 No irq handler for vector [ 945.692058] do_IRQ: 0.41 No irq handler for vector [ 945.694878] do_IRQ: 0.41 No irq handler for vector [ 945.704593] do_IRQ: 0.41 No irq handler for vector [ 945.708540] do_IRQ: 0.41 No irq handler for vector [ 945.716677] do_IRQ: 0.41 No irq handler for vector [ 950.761664] LustreError: 14695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 950.780579] do_IRQ: 1 callbacks suppressed [ 950.780590] do_IRQ: 0.41 No irq handler for vector [ 950.791511] do_IRQ: 0.41 No irq handler for vector [ 950.796693] LustreError: 14695:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 950.815820] do_IRQ: 0.41 No irq handler for vector [ 950.819129] do_IRQ: 0.41 No irq handler for vector [ 950.822600] do_IRQ: 0.41 No irq handler for vector [ 950.824889] do_IRQ: 0.41 No irq handler for vector [ 950.828644] do_IRQ: 0.41 No irq handler for vector [ 950.831762] do_IRQ: 0.41 No irq handler for vector [ 950.836270] do_IRQ: 0.41 No irq handler for vector [ 950.839461] do_IRQ: 0.41 No irq handler for vector [ 954.716655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 954.979737] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 955.004825] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 956.605637] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 956.611431] do_IRQ: 4 callbacks suppressed [ 956.611442] do_IRQ: 0.41 No irq handler for vector [ 956.625747] do_IRQ: 0.41 No irq handler for vector [ 956.629883] do_IRQ: 0.41 No irq handler for vector [ 956.633994] do_IRQ: 0.41 No irq handler for vector [ 956.644880] do_IRQ: 0.41 No irq handler for vector [ 956.648440] do_IRQ: 0.41 No irq handler for vector [ 956.657774] do_IRQ: 0.41 No irq handler for vector [ 956.660843] do_IRQ: 0.41 No irq handler for vector [ 956.676295] do_IRQ: 0.41 No irq handler for vector [ 956.679831] do_IRQ: 0.41 No irq handler for vector [ 956.931262] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 956.932029] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 961.704809] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [ 961.710561] do_IRQ: 3 callbacks suppressed [ 961.710572] do_IRQ: 0.41 No irq handler for vector [ 961.723043] do_IRQ: 0.41 No irq handler for vector [ 961.729609] do_IRQ: 0.41 No irq handler for vector [ 961.731841] do_IRQ: 0.41 No irq handler for vector [ 961.735797] do_IRQ: 0.41 No irq handler for vector [ 961.739035] do_IRQ: 0.41 No irq handler for vector [ 961.740581] do_IRQ: 0.41 No irq handler for vector [ 961.743876] do_IRQ: 0.41 No irq handler for vector [ 961.746882] do_IRQ: 0.41 No irq handler for vector [ 961.750266] do_IRQ: 0.41 No irq handler for vector [ 970.695889] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 970.708999] do_IRQ: 1 callbacks suppressed [ 970.709011] do_IRQ: 0.41 No irq handler for vector [ 970.730495] do_IRQ: 0.41 No irq handler for vector [ 970.735518] do_IRQ: 0.41 No irq handler for vector [ 970.746589] do_IRQ: 0.41 No irq handler for vector [ 970.749120] do_IRQ: 0.41 No irq handler for vector [ 970.757950] do_IRQ: 0.41 No irq handler for vector [ 970.764925] do_IRQ: 0.41 No irq handler for vector [ 970.771215] do_IRQ: 0.41 No irq handler for vector [ 970.776150] do_IRQ: 0.41 No irq handler for vector [ 970.779374] do_IRQ: 0.41 No irq handler for vector [ 972.254611] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 985.354633] Lustre: DEBUG MARKER: == sanity test 64j: check grants on re-done rpc ========== 00:08:14 (1790741294) [ 985.364937] do_IRQ: 2 callbacks suppressed [ 985.364957] do_IRQ: 0.41 No irq handler for vector [ 985.369167] do_IRQ: 0.41 No irq handler for vector [ 985.371687] do_IRQ: 0.41 No irq handler for vector [ 985.373819] do_IRQ: 0.41 No irq handler for vector [ 985.378121] do_IRQ: 0.41 No irq handler for vector [ 985.380768] do_IRQ: 0.41 No irq handler for vector [ 985.383972] do_IRQ: 0.41 No irq handler for vector [ 985.389325] do_IRQ: 0.41 No irq handler for vector [ 985.392862] do_IRQ: 0.41 No irq handler for vector [ 985.399575] do_IRQ: 0.41 No irq handler for vector [ 986.546304] Lustre: *** cfs_fail_loc=256, val=0*** [ 996.167540] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 00:08:25 (1790741305) [ 996.175290] do_IRQ: 3 callbacks suppressed [ 996.175301] do_IRQ: 0.41 No irq handler for vector [ 996.179713] do_IRQ: 0.41 No irq handler for vector [ 996.186158] do_IRQ: 0.41 No irq handler for vector [ 996.196385] do_IRQ: 0.41 No irq handler for vector [ 996.204076] do_IRQ: 0.41 No irq handler for vector [ 996.208373] do_IRQ: 0.41 No irq handler for vector [ 996.212279] do_IRQ: 0.41 No irq handler for vector [ 996.214934] do_IRQ: 0.41 No irq handler for vector [ 996.220235] do_IRQ: 0.41 No irq handler for vector [ 996.223354] do_IRQ: 0.41 No irq handler for vector [ 1003.045418] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 00:08:32 (1790741312) [ 1003.057041] do_IRQ: 2 callbacks suppressed [ 1003.057050] do_IRQ: 0.41 No irq handler for vector [ 1003.059630] do_IRQ: 0.41 No irq handler for vector [ 1003.061978] do_IRQ: 0.41 No irq handler for vector [ 1003.065449] do_IRQ: 0.41 No irq handler for vector [ 1003.068209] do_IRQ: 0.41 No irq handler for vector [ 1003.072049] do_IRQ: 0.41 No irq handler for vector [ 1003.074825] do_IRQ: 0.41 No irq handler for vector [ 1003.079743] do_IRQ: 0.41 No irq handler for vector [ 1003.082498] do_IRQ: 0.41 No irq handler for vector [ 1003.085413] do_IRQ: 0.41 No irq handler for vector [ 1010.744514] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 00:08:39 (1790741319) [ 1010.762679] do_IRQ: 2 callbacks suppressed [ 1010.762687] do_IRQ: 0.41 No irq handler for vector [ 1010.765446] do_IRQ: 0.41 No irq handler for vector [ 1010.768993] do_IRQ: 0.41 No irq handler for vector [ 1010.772417] do_IRQ: 0.41 No irq handler for vector [ 1010.779313] do_IRQ: 0.41 No irq handler for vector [ 1010.782357] do_IRQ: 0.41 No irq handler for vector [ 1010.785212] do_IRQ: 0.41 No irq handler for vector [ 1010.787573] do_IRQ: 0.41 No irq handler for vector [ 1010.795925] do_IRQ: 0.41 No irq handler for vector [ 1010.800739] do_IRQ: 0.41 No irq handler for vector [ 1017.735645] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 00:08:46 (1790741326) [ 1017.753359] do_IRQ: 1 callbacks suppressed [ 1017.753369] do_IRQ: 0.41 No irq handler for vector [ 1017.765621] do_IRQ: 0.41 No irq handler for vector [ 1017.772769] do_IRQ: 0.41 No irq handler for vector [ 1017.775884] do_IRQ: 0.41 No irq handler for vector [ 1017.779236] do_IRQ: 0.41 No irq handler for vector [ 1017.783390] do_IRQ: 0.41 No irq handler for vector [ 1017.786599] do_IRQ: 0.41 No irq handler for vector [ 1017.789743] do_IRQ: 0.41 No irq handler for vector [ 1017.792985] do_IRQ: 0.41 No irq handler for vector [ 1017.794949] do_IRQ: 0.41 No irq handler for vector [ 1025.171568] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 00:08:54 (1790741334) [ 1025.178702] do_IRQ: 1 callbacks suppressed [ 1025.178713] do_IRQ: 0.41 No irq handler for vector [ 1025.181625] do_IRQ: 0.41 No irq handler for vector [ 1025.184939] do_IRQ: 0.41 No irq handler for vector [ 1025.191517] do_IRQ: 0.41 No irq handler for vector [ 1025.194421] do_IRQ: 0.41 No irq handler for vector [ 1025.197505] do_IRQ: 0.41 No irq handler for vector [ 1025.200412] do_IRQ: 0.41 No irq handler for vector [ 1025.204027] do_IRQ: 0.41 No irq handler for vector [ 1025.212120] do_IRQ: 0.41 No irq handler for vector [ 1025.214785] do_IRQ: 0.41 No irq handler for vector [ 1032.247247] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 00:09:01 (1790741341) [ 1032.265626] do_IRQ: 2 callbacks suppressed [ 1032.265637] do_IRQ: 0.41 No irq handler for vector [ 1032.270802] do_IRQ: 0.41 No irq handler for vector [ 1032.273488] do_IRQ: 0.41 No irq handler for vector [ 1032.279239] do_IRQ: 0.41 No irq handler for vector [ 1032.291396] do_IRQ: 0.41 No irq handler for vector [ 1032.296550] do_IRQ: 0.41 No irq handler for vector [ 1032.300751] do_IRQ: 0.41 No irq handler for vector [ 1032.306860] do_IRQ: 0.41 No irq handler for vector [ 1032.310728] do_IRQ: 0.41 No irq handler for vector [ 1032.314075] do_IRQ: 0.41 No irq handler for vector [ 1039.903944] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 00:09:08 (1790741348) [ 1039.909366] do_IRQ: 2 callbacks suppressed [ 1039.909376] do_IRQ: 0.41 No irq handler for vector [ 1039.917018] do_IRQ: 0.41 No irq handler for vector [ 1039.920145] do_IRQ: 0.41 No irq handler for vector [ 1039.922466] do_IRQ: 0.41 No irq handler for vector [ 1039.926309] do_IRQ: 0.41 No irq handler for vector [ 1039.930873] do_IRQ: 0.41 No irq handler for vector [ 1039.932892] do_IRQ: 0.41 No irq handler for vector [ 1039.936369] do_IRQ: 0.41 No irq handler for vector [ 1039.941769] do_IRQ: 0.41 No irq handler for vector [ 1039.948472] do_IRQ: 0.41 No irq handler for vector [ 1047.671415] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 00:09:16 (1790741356) [ 1047.687475] do_IRQ: 1 callbacks suppressed [ 1047.687485] do_IRQ: 0.41 No irq handler for vector [ 1047.698839] do_IRQ: 0.41 No irq handler for vector [ 1047.706375] do_IRQ: 0.41 No irq handler for vector [ 1047.715321] do_IRQ: 0.41 No irq handler for vector [ 1047.717070] do_IRQ: 0.41 No irq handler for vector [ 1047.728750] do_IRQ: 0.41 No irq handler for vector [ 1047.736534] do_IRQ: 0.41 No irq handler for vector [ 1047.739986] do_IRQ: 0.41 No irq handler for vector [ 1047.747300] do_IRQ: 0.41 No irq handler for vector [ 1047.756426] do_IRQ: 0.41 No irq handler for vector [ 1055.930925] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 00:09:24 (1790741364) [ 1055.941379] do_IRQ: 1 callbacks suppressed [ 1055.941389] do_IRQ: 0.41 No irq handler for vector [ 1055.954915] do_IRQ: 0.41 No irq handler for vector [ 1055.958596] do_IRQ: 0.41 No irq handler for vector [ 1055.964482] do_IRQ: 0.41 No irq handler for vector [ 1055.972120] do_IRQ: 0.41 No irq handler for vector [ 1055.973550] do_IRQ: 0.41 No irq handler for vector [ 1055.976662] do_IRQ: 0.41 No irq handler for vector [ 1055.981888] do_IRQ: 0.41 No irq handler for vector [ 1055.987552] do_IRQ: 0.41 No irq handler for vector [ 1055.992761] do_IRQ: 0.41 No irq handler for vector [ 1064.196403] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 00:09:33 (1790741373) [ 1064.215883] do_IRQ: 1 callbacks suppressed [ 1064.215895] do_IRQ: 0.41 No irq handler for vector [ 1064.235823] do_IRQ: 0.41 No irq handler for vector [ 1064.248722] do_IRQ: 0.41 No irq handler for vector [ 1064.253501] do_IRQ: 0.41 No irq handler for vector [ 1064.257294] do_IRQ: 0.41 No irq handler for vector [ 1064.261929] do_IRQ: 0.41 No irq handler for vector [ 1064.265568] do_IRQ: 0.41 No irq handler for vector [ 1064.271335] do_IRQ: 0.41 No irq handler for vector [ 1064.278853] do_IRQ: 0.41 No irq handler for vector [ 1064.282957] do_IRQ: 0.41 No irq handler for vector [ 1071.705229] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 00:09:40 (1790741380) [ 1071.716040] do_IRQ: 1 callbacks suppressed [ 1071.716051] do_IRQ: 0.41 No irq handler for vector [ 1071.719136] do_IRQ: 0.41 No irq handler for vector [ 1071.721062] do_IRQ: 0.41 No irq handler for vector [ 1071.724864] do_IRQ: 0.41 No irq handler for vector [ 1071.727743] do_IRQ: 0.41 No irq handler for vector [ 1071.731278] do_IRQ: 0.41 No irq handler for vector [ 1071.734498] do_IRQ: 0.41 No irq handler for vector [ 1071.736666] do_IRQ: 0.41 No irq handler for vector [ 1071.738221] do_IRQ: 0.41 No irq handler for vector [ 1071.742911] do_IRQ: 0.41 No irq handler for vector [ 1073.557590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1073.573818] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 1073.590804] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1073.597543] Lustre: Skipped 1 previous similar message [ 1074.299742] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1075.096952] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.111203] Lustre: Skipped 1 previous similar message [ 1075.116442] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1075.123192] Lustre: Skipped 1 previous similar message [ 1075.900976] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1075.908194] Lustre: Skipped 1 previous similar message [ 1092.852964] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1092.856571] do_IRQ: 13 callbacks suppressed [ 1092.856582] do_IRQ: 0.41 No irq handler for vector [ 1092.867981] do_IRQ: 0.41 No irq handler for vector [ 1092.874336] do_IRQ: 0.41 No irq handler for vector [ 1092.876336] do_IRQ: 0.41 No irq handler for vector [ 1092.879078] do_IRQ: 0.41 No irq handler for vector [ 1092.882056] do_IRQ: 0.41 No irq handler for vector [ 1092.888231] do_IRQ: 0.41 No irq handler for vector [ 1092.894932] do_IRQ: 0.41 No irq handler for vector [ 1092.899816] do_IRQ: 0.41 No irq handler for vector [ 1092.909275] do_IRQ: 0.41 No irq handler for vector [ 1107.636679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.665833] do_IRQ: 1 callbacks suppressed [ 1107.665845] do_IRQ: 0.41 No irq handler for vector [ 1107.671481] do_IRQ: 0.41 No irq handler for vector [ 1107.680506] do_IRQ: 0.41 No irq handler for vector [ 1107.684085] do_IRQ: 0.41 No irq handler for vector [ 1107.686913] do_IRQ: 0.41 No irq handler for vector [ 1107.689785] do_IRQ: 0.41 No irq handler for vector [ 1107.695658] do_IRQ: 0.41 No irq handler for vector [ 1107.697746] do_IRQ: 0.41 No irq handler for vector [ 1107.700976] do_IRQ: 0.41 No irq handler for vector [ 1107.707917] do_IRQ: 0.41 No irq handler for vector [ 1107.712839] Lustre: Skipped 1 previous similar message [ 1107.718355] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 1107.734189] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1107.753772] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1107.762230] Lustre: Skipped 1 previous similar message [ 1111.929931] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1112.133794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1116.774451] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1116.783225] do_IRQ: 9 callbacks suppressed [ 1116.783235] do_IRQ: 0.41 No irq handler for vector [ 1116.796820] do_IRQ: 0.41 No irq handler for vector [ 1116.800493] do_IRQ: 0.41 No irq handler for vector [ 1116.807252] do_IRQ: 0.41 No irq handler for vector [ 1116.812546] do_IRQ: 0.41 No irq handler for vector [ 1116.817484] do_IRQ: 0.41 No irq handler for vector [ 1116.821435] do_IRQ: 0.41 No irq handler for vector [ 1116.824989] do_IRQ: 0.41 No irq handler for vector [ 1116.828905] do_IRQ: 0.41 No irq handler for vector [ 1116.831945] do_IRQ: 0.41 No irq handler for vector [ 1117.066804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1133.001117] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1133.005020] do_IRQ: 2 callbacks suppressed [ 1133.005032] do_IRQ: 0.41 No irq handler for vector [ 1133.016690] do_IRQ: 0.41 No irq handler for vector [ 1133.019750] do_IRQ: 0.41 No irq handler for vector [ 1133.024522] do_IRQ: 0.41 No irq handler for vector [ 1133.028845] do_IRQ: 0.41 No irq handler for vector [ 1133.031693] do_IRQ: 0.41 No irq handler for vector [ 1133.035872] do_IRQ: 0.41 No irq handler for vector [ 1133.038301] do_IRQ: 0.41 No irq handler for vector [ 1133.044319] do_IRQ: 0.41 No irq handler for vector [ 1133.047141] do_IRQ: 0.41 No irq handler for vector [ 1147.167640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1147.181947] do_IRQ: 1 callbacks suppressed [ 1147.181959] do_IRQ: 0.41 No irq handler for vector [ 1147.185949] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1147.189199] do_IRQ: 0.41 No irq handler for vector [ 1147.194247] do_IRQ: 0.41 No irq handler for vector [ 1147.202666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1147.203986] do_IRQ: 0.41 No irq handler for vector [ 1147.220784] do_IRQ: 0.41 No irq handler for vector [ 1147.225817] do_IRQ: 0.41 No irq handler for vector [ 1147.228298] do_IRQ: 0.41 No irq handler for vector [ 1147.230908] do_IRQ: 0.41 No irq handler for vector [ 1147.240200] do_IRQ: 0.41 No irq handler for vector [ 1147.246031] do_IRQ: 0.41 No irq handler for vector [ 1147.255189] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1151.098673] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1151.264518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1155.204580] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1155.212922] do_IRQ: 4 callbacks suppressed [ 1155.212931] do_IRQ: 0.41 No irq handler for vector [ 1155.221904] do_IRQ: 0.41 No irq handler for vector [ 1155.227807] do_IRQ: 0.41 No irq handler for vector [ 1155.232213] do_IRQ: 0.41 No irq handler for vector [ 1155.236586] do_IRQ: 0.41 No irq handler for vector [ 1155.241969] do_IRQ: 0.41 No irq handler for vector [ 1155.246773] do_IRQ: 0.41 No irq handler for vector [ 1155.249963] do_IRQ: 0.41 No irq handler for vector [ 1155.253540] do_IRQ: 0.41 No irq handler for vector [ 1155.255944] do_IRQ: 0.41 No irq handler for vector [ 1155.470746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1173.673940] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1173.680730] do_IRQ: 2 callbacks suppressed [ 1173.680737] do_IRQ: 0.41 No irq handler for vector [ 1173.693707] do_IRQ: 0.41 No irq handler for vector [ 1173.696352] do_IRQ: 0.41 No irq handler for vector [ 1173.700118] do_IRQ: 0.41 No irq handler for vector [ 1173.703553] do_IRQ: 0.41 No irq handler for vector [ 1173.708414] do_IRQ: 0.41 No irq handler for vector [ 1173.716905] do_IRQ: 0.41 No irq handler for vector [ 1173.720782] do_IRQ: 0.41 No irq handler for vector [ 1173.723528] do_IRQ: 0.41 No irq handler for vector [ 1173.730030] do_IRQ: 0.41 No irq handler for vector [ 1188.983301] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.989880] do_IRQ: 1 callbacks suppressed [ 1188.989896] do_IRQ: 0.41 No irq handler for vector [ 1188.993848] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 1188.997062] do_IRQ: 0.41 No irq handler for vector [ 1189.001820] do_IRQ: 0.41 No irq handler for vector [ 1189.012724] do_IRQ: 0.41 No irq handler for vector [ 1189.016231] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1189.019001] do_IRQ: 0.41 No irq handler for vector [ 1189.024599] do_IRQ: 0.41 No irq handler for vector [ 1189.042064] do_IRQ: 0.41 No irq handler for vector [ 1189.045120] do_IRQ: 0.41 No irq handler for vector [ 1189.047785] do_IRQ: 0.41 No irq handler for vector [ 1189.050685] do_IRQ: 0.41 No irq handler for vector [ 1189.055863] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1193.477770] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1193.682087] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1198.392506] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1198.399935] do_IRQ: 4 callbacks suppressed [ 1198.399945] do_IRQ: 0.41 No irq handler for vector [ 1198.416549] do_IRQ: 0.41 No irq handler for vector [ 1198.420291] do_IRQ: 0.41 No irq handler for vector [ 1198.425011] do_IRQ: 0.41 No irq handler for vector [ 1198.428491] do_IRQ: 0.41 No irq handler for vector [ 1198.431294] do_IRQ: 0.41 No irq handler for vector [ 1198.433141] do_IRQ: 0.41 No irq handler for vector [ 1198.434550] do_IRQ: 0.41 No irq handler for vector [ 1198.439688] do_IRQ: 0.41 No irq handler for vector [ 1198.442383] do_IRQ: 0.41 No irq handler for vector [ 1198.601264] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1214.820648] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1214.823918] do_IRQ: 2 callbacks suppressed [ 1214.823929] do_IRQ: 0.41 No irq handler for vector [ 1214.830064] do_IRQ: 0.41 No irq handler for vector [ 1214.832283] do_IRQ: 0.41 No irq handler for vector [ 1214.836858] do_IRQ: 0.41 No irq handler for vector [ 1214.842794] do_IRQ: 0.41 No irq handler for vector [ 1214.847276] do_IRQ: 0.41 No irq handler for vector [ 1214.851410] do_IRQ: 0.41 No irq handler for vector [ 1214.854471] do_IRQ: 0.41 No irq handler for vector [ 1214.859857] do_IRQ: 0.41 No irq handler for vector [ 1214.863332] do_IRQ: 0.41 No irq handler for vector [ 1228.144101] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.150534] do_IRQ: 1 callbacks suppressed [ 1228.150542] do_IRQ: 0.41 No irq handler for vector [ 1228.152907] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1228.153686] do_IRQ: 0.41 No irq handler for vector [ 1228.156310] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1228.159147] do_IRQ: 0.41 No irq handler for vector [ 1228.160546] do_IRQ: 0.41 No irq handler for vector [ 1228.165991] do_IRQ: 0.41 No irq handler for vector [ 1228.172441] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1228.173861] do_IRQ: 0.41 No irq handler for vector [ 1228.176500] do_IRQ: 0.41 No irq handler for vector [ 1228.178410] do_IRQ: 0.41 No irq handler for vector [ 1228.183820] do_IRQ: 0.41 No irq handler for vector [ 1228.189312] do_IRQ: 0.41 No irq handler for vector [ 1231.705340] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1231.958993] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1236.338206] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1236.343471] do_IRQ: 3 callbacks suppressed [ 1236.343483] do_IRQ: 0.41 No irq handler for vector [ 1236.353302] do_IRQ: 0.41 No irq handler for vector [ 1236.365440] do_IRQ: 0.41 No irq handler for vector [ 1236.367508] do_IRQ: 0.41 No irq handler for vector [ 1236.373968] do_IRQ: 0.41 No irq handler for vector [ 1236.377797] do_IRQ: 0.41 No irq handler for vector [ 1236.381425] do_IRQ: 0.41 No irq handler for vector [ 1236.383598] do_IRQ: 0.41 No irq handler for vector [ 1236.386724] do_IRQ: 0.41 No irq handler for vector [ 1236.388295] do_IRQ: 0.41 No irq handler for vector [ 1236.537780] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1242.681729] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 00:12:32 (1790741552) [ 1242.695242] do_IRQ: 2 callbacks suppressed [ 1242.695250] do_IRQ: 0.41 No irq handler for vector [ 1242.701237] do_IRQ: 0.41 No irq handler for vector [ 1242.703736] do_IRQ: 0.41 No irq handler for vector [ 1242.706683] do_IRQ: 0.41 No irq handler for vector [ 1242.709342] do_IRQ: 0.41 No irq handler for vector [ 1242.711499] do_IRQ: 0.41 No irq handler for vector [ 1242.713664] do_IRQ: 0.41 No irq handler for vector [ 1242.715360] do_IRQ: 0.41 No irq handler for vector [ 1242.719578] do_IRQ: 0.41 No irq handler for vector [ 1242.722755] do_IRQ: 0.41 No irq handler for vector [ 1249.075236] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 00:12:38 (1790741558) [ 1249.089374] do_IRQ: 1 callbacks suppressed [ 1249.089384] do_IRQ: 0.41 No irq handler for vector [ 1249.093411] do_IRQ: 0.41 No irq handler for vector [ 1249.098346] do_IRQ: 0.41 No irq handler for vector [ 1249.105540] do_IRQ: 0.41 No irq handler for vector [ 1249.108930] do_IRQ: 0.41 No irq handler for vector [ 1249.115289] do_IRQ: 0.41 No irq handler for vector [ 1249.121836] do_IRQ: 0.41 No irq handler for vector [ 1249.128175] do_IRQ: 0.41 No irq handler for vector [ 1249.132384] do_IRQ: 0.41 No irq handler for vector [ 1249.138522] do_IRQ: 0.41 No irq handler for vector [ 1255.346605] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 00:12:44 (1790741564) [ 1255.358150] do_IRQ: 2 callbacks suppressed [ 1255.358158] do_IRQ: 0.41 No irq handler for vector [ 1255.364321] do_IRQ: 0.41 No irq handler for vector [ 1255.365971] do_IRQ: 0.41 No irq handler for vector [ 1255.373695] do_IRQ: 0.41 No irq handler for vector [ 1255.381013] do_IRQ: 0.41 No irq handler for vector [ 1255.385899] do_IRQ: 0.41 No irq handler for vector [ 1255.389342] do_IRQ: 0.41 No irq handler for vector [ 1255.397209] do_IRQ: 0.41 No irq handler for vector [ 1255.399024] do_IRQ: 0.41 No irq handler for vector [ 1255.409935] do_IRQ: 0.41 No irq handler for vector [ 1285.350694] Lustre: DEBUG MARKER: == sanity test 65o: pool inheritance for mdt component === 00:13:14 (1790741594) [ 1285.371544] do_IRQ: 1 callbacks suppressed [ 1285.371553] do_IRQ: 0.41 No irq handler for vector [ 1285.384716] do_IRQ: 0.41 No irq handler for vector [ 1285.388539] do_IRQ: 0.41 No irq handler for vector [ 1285.392378] do_IRQ: 0.41 No irq handler for vector [ 1285.400711] do_IRQ: 0.41 No irq handler for vector [ 1285.412815] do_IRQ: 0.41 No irq handler for vector [ 1285.414619] do_IRQ: 0.41 No irq handler for vector [ 1285.420967] do_IRQ: 0.41 No irq handler for vector [ 1285.426597] do_IRQ: 0.41 No irq handler for vector [ 1285.430979] do_IRQ: 0.41 No irq handler for vector [ 1313.430628] Lustre: DEBUG MARKER: == sanity test 65p: setstripe with yaml file and huge number ========================================================== 00:13:42 (1790741622) [ 1313.443854] do_IRQ: 1 callbacks suppressed [ 1313.443934] do_IRQ: 0.41 No irq handler for vector [ 1313.446706] do_IRQ: 0.41 No irq handler for vector [ 1313.449497] do_IRQ: 0.41 No irq handler for vector [ 1313.457212] do_IRQ: 0.41 No irq handler for vector [ 1313.462467] do_IRQ: 0.41 No irq handler for vector [ 1313.465246] do_IRQ: 0.41 No irq handler for vector [ 1313.468485] do_IRQ: 0.41 No irq handler for vector [ 1313.470958] do_IRQ: 0.41 No irq handler for vector [ 1313.473691] do_IRQ: 0.41 No irq handler for vector [ 1313.475958] do_IRQ: 0.41 No irq handler for vector [ 1321.025535] Lustre: DEBUG MARKER: == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 00:13:50 (1790741630) [ 1321.035708] do_IRQ: 2 callbacks suppressed [ 1321.035721] do_IRQ: 0.41 No irq handler for vector [ 1321.049087] do_IRQ: 0.41 No irq handler for vector [ 1321.054729] do_IRQ: 0.41 No irq handler for vector [ 1321.060024] do_IRQ: 0.41 No irq handler for vector [ 1321.068382] do_IRQ: 0.41 No irq handler for vector [ 1321.074339] do_IRQ: 0.41 No irq handler for vector [ 1321.076836] do_IRQ: 0.41 No irq handler for vector [ 1321.080320] do_IRQ: 0.41 No irq handler for vector [ 1321.083959] do_IRQ: 0.41 No irq handler for vector [ 1321.091258] do_IRQ: 0.41 No irq handler for vector [ 1327.809370] Lustre: DEBUG MARKER: == sanity test 65r: prevent all-zero offsets ============= 00:13:56 (1790741636) [ 1327.815115] do_IRQ: 1 callbacks suppressed [ 1327.815187] do_IRQ: 0.41 No irq handler for vector [ 1327.818221] do_IRQ: 0.41 No irq handler for vector [ 1327.823884] do_IRQ: 0.41 No irq handler for vector [ 1327.827770] do_IRQ: 0.41 No irq handler for vector [ 1327.832948] do_IRQ: 0.41 No irq handler for vector [ 1327.836597] do_IRQ: 0.41 No irq handler for vector [ 1327.843246] do_IRQ: 0.41 No irq handler for vector [ 1327.854553] do_IRQ: 0.41 No irq handler for vector [ 1327.862376] do_IRQ: 0.41 No irq handler for vector [ 1327.868499] do_IRQ: 0.41 No irq handler for vector [ 1334.866738] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 00:14:04 (1790741644) [ 1334.876264] do_IRQ: 2 callbacks suppressed [ 1334.876276] do_IRQ: 0.41 No irq handler for vector [ 1334.886574] do_IRQ: 0.41 No irq handler for vector [ 1334.894299] do_IRQ: 0.41 No irq handler for vector [ 1334.896966] do_IRQ: 0.41 No irq handler for vector [ 1334.910249] do_IRQ: 0.41 No irq handler for vector [ 1334.911348] do_IRQ: 0.41 No irq handler for vector [ 1334.912453] do_IRQ: 0.41 No irq handler for vector [ 1334.922467] do_IRQ: 0.41 No irq handler for vector [ 1334.932605] do_IRQ: 0.41 No irq handler for vector [ 1334.936458] do_IRQ: 0.41 No irq handler for vector [ 1347.155758] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 00:14:16 (1790741656) [ 1347.169476] do_IRQ: 1 callbacks suppressed [ 1347.169488] do_IRQ: 0.41 No irq handler for vector [ 1347.175642] do_IRQ: 0.41 No irq handler for vector [ 1347.177231] do_IRQ: 0.41 No irq handler for vector [ 1347.182603] do_IRQ: 0.41 No irq handler for vector [ 1347.186715] do_IRQ: 0.41 No irq handler for vector [ 1347.190793] do_IRQ: 0.41 No irq handler for vector [ 1347.196204] do_IRQ: 0.41 No irq handler for vector [ 1347.200796] do_IRQ: 0.41 No irq handler for vector [ 1347.207547] do_IRQ: 0.41 No irq handler for vector [ 1347.221691] do_IRQ: 0.41 No irq handler for vector [ 1348.645513] Lustre: *** cfs_fail_loc=217, val=0*** [ 1350.786038] Lustre: *** cfs_fail_loc=217, val=0*** [ 1350.787468] Lustre: Skipped 1 previous similar message [ 1357.572900] Lustre: DEBUG MARKER: == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 00:14:26 (1790741666) [ 1357.589005] do_IRQ: 4 callbacks suppressed [ 1357.589015] do_IRQ: 0.41 No irq handler for vector [ 1357.608581] do_IRQ: 0.41 No irq handler for vector [ 1357.618226] do_IRQ: 0.41 No irq handler for vector [ 1357.624100] do_IRQ: 0.41 No irq handler for vector [ 1357.631088] do_IRQ: 0.41 No irq handler for vector [ 1357.637668] do_IRQ: 0.41 No irq handler for vector [ 1357.651349] do_IRQ: 0.41 No irq handler for vector [ 1357.659873] do_IRQ: 0.41 No irq handler for vector [ 1357.665607] do_IRQ: 0.41 No irq handler for vector [ 1357.674433] do_IRQ: 0.41 No irq handler for vector [ 1370.007811] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 1370.018459] do_IRQ: 1 callbacks suppressed [ 1370.018469] do_IRQ: 0.41 No irq handler for vector [ 1370.026234] do_IRQ: 0.41 No irq handler for vector [ 1370.036858] do_IRQ: 0.41 No irq handler for vector [ 1370.038591] do_IRQ: 0.41 No irq handler for vector [ 1370.044829] do_IRQ: 0.41 No irq handler for vector [ 1370.051017] do_IRQ: 0.41 No irq handler for vector [ 1370.052702] do_IRQ: 0.41 No irq handler for vector [ 1370.057028] do_IRQ: 0.41 No irq handler for vector [ 1370.061710] do_IRQ: 0.41 No irq handler for vector [ 1370.064711] do_IRQ: 0.41 No irq handler for vector [ 1371.703024] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 00:14:40 (1790741680) [ 1378.503733] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 00:14:47 (1790741687) [ 1378.511343] do_IRQ: 2 callbacks suppressed [ 1378.511354] do_IRQ: 0.41 No irq handler for vector [ 1378.520467] do_IRQ: 0.41 No irq handler for vector [ 1378.524489] do_IRQ: 0.41 No irq handler for vector [ 1378.530179] do_IRQ: 0.41 No irq handler for vector [ 1378.532839] do_IRQ: 0.41 No irq handler for vector [ 1378.539751] do_IRQ: 0.41 No irq handler for vector [ 1378.543629] do_IRQ: 0.41 No irq handler for vector [ 1378.546176] do_IRQ: 0.41 No irq handler for vector [ 1378.553653] do_IRQ: 0.41 No irq handler for vector [ 1378.559809] do_IRQ: 0.41 No irq handler for vector [ 1385.874340] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 00:14:55 (1790741695) [ 1385.896410] do_IRQ: 1 callbacks suppressed [ 1385.896420] do_IRQ: 0.41 No irq handler for vector [ 1385.904467] do_IRQ: 0.41 No irq handler for vector [ 1385.918035] do_IRQ: 0.41 No irq handler for vector [ 1385.924241] do_IRQ: 0.41 No irq handler for vector [ 1385.925535] do_IRQ: 0.41 No irq handler for vector [ 1385.931180] do_IRQ: 0.41 No irq handler for vector [ 1385.935539] do_IRQ: 0.41 No irq handler for vector [ 1385.937404] do_IRQ: 0.41 No irq handler for vector [ 1385.945727] do_IRQ: 0.41 No irq handler for vector [ 1385.950291] do_IRQ: 0.41 No irq handler for vector [ 1419.376673] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 00:15:28 (1790741728) [ 1419.388954] do_IRQ: 1 callbacks suppressed [ 1419.388965] do_IRQ: 0.41 No irq handler for vector [ 1419.391775] do_IRQ: 0.41 No irq handler for vector [ 1419.394541] do_IRQ: 0.41 No irq handler for vector [ 1419.396740] do_IRQ: 0.41 No irq handler for vector [ 1419.401542] do_IRQ: 0.41 No irq handler for vector [ 1419.404631] do_IRQ: 0.41 No irq handler for vector [ 1419.410178] do_IRQ: 0.41 No irq handler for vector [ 1419.415156] do_IRQ: 0.41 No irq handler for vector [ 1419.418806] do_IRQ: 0.41 No irq handler for vector [ 1419.423751] do_IRQ: 0.41 No irq handler for vector [ 1425.112342] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 00:15:34 (1790741734) [ 1425.119777] do_IRQ: 2 callbacks suppressed [ 1425.119789] do_IRQ: 0.41 No irq handler for vector [ 1425.126838] do_IRQ: 0.41 No irq handler for vector [ 1425.128563] do_IRQ: 0.41 No irq handler for vector [ 1425.132298] do_IRQ: 0.41 No irq handler for vector [ 1425.134339] do_IRQ: 0.41 No irq handler for vector [ 1425.142068] do_IRQ: 0.41 No irq handler for vector [ 1425.145051] do_IRQ: 0.41 No irq handler for vector [ 1425.149606] do_IRQ: 0.41 No irq handler for vector [ 1425.154192] do_IRQ: 0.41 No irq handler for vector [ 1425.161384] do_IRQ: 0.41 No irq handler for vector [ 1430.984923] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 00:15:40 (1790741740) [ 1431.011802] do_IRQ: 1 callbacks suppressed [ 1431.011809] do_IRQ: 0.41 No irq handler for vector [ 1431.019698] do_IRQ: 0.41 No irq handler for vector [ 1431.026646] do_IRQ: 0.41 No irq handler for vector [ 1431.029487] do_IRQ: 0.41 No irq handler for vector [ 1431.034093] do_IRQ: 0.41 No irq handler for vector [ 1431.039154] do_IRQ: 0.41 No irq handler for vector [ 1431.041710] do_IRQ: 0.41 No irq handler for vector [ 1431.048578] do_IRQ: 0.41 No irq handler for vector [ 1431.056922] do_IRQ: 0.41 No irq handler for vector [ 1431.062086] do_IRQ: 0.41 No irq handler for vector [ 1437.145913] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 00:15:46 (1790741746) [ 1437.158261] do_IRQ: 1 callbacks suppressed [ 1437.158269] do_IRQ: 0.41 No irq handler for vector [ 1437.164760] do_IRQ: 0.41 No irq handler for vector [ 1437.173584] do_IRQ: 0.41 No irq handler for vector [ 1437.185278] do_IRQ: 0.41 No irq handler for vector [ 1437.191773] do_IRQ: 0.41 No irq handler for vector [ 1437.193301] do_IRQ: 0.41 No irq handler for vector [ 1437.201148] do_IRQ: 0.41 No irq handler for vector [ 1437.206453] do_IRQ: 0.41 No irq handler for vector [ 1437.211941] do_IRQ: 0.41 No irq handler for vector [ 1437.214774] do_IRQ: 0.41 No irq handler for vector [ 1508.491601] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 00:16:57 (1790741817) [ 1508.502927] do_IRQ: 1 callbacks suppressed [ 1508.502940] do_IRQ: 0.41 No irq handler for vector [ 1508.516847] do_IRQ: 0.41 No irq handler for vector [ 1508.520199] do_IRQ: 0.41 No irq handler for vector [ 1508.526382] do_IRQ: 0.41 No irq handler for vector [ 1508.535426] do_IRQ: 0.41 No irq handler for vector [ 1508.538077] do_IRQ: 0.41 No irq handler for vector [ 1508.543971] do_IRQ: 0.41 No irq handler for vector [ 1508.547434] do_IRQ: 0.41 No irq handler for vector [ 1508.556416] do_IRQ: 0.41 No irq handler for vector [ 1508.563355] do_IRQ: 0.41 No irq handler for vector [ 1560.398995] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 00:17:49 (1790741869) [ 1560.415888] do_IRQ: 1 callbacks suppressed [ 1560.415899] do_IRQ: 0.41 No irq handler for vector [ 1560.426097] do_IRQ: 0.41 No irq handler for vector [ 1560.431637] do_IRQ: 0.41 No irq handler for vector [ 1560.436143] do_IRQ: 0.41 No irq handler for vector [ 1560.440112] do_IRQ: 0.41 No irq handler for vector [ 1560.449205] do_IRQ: 0.41 No irq handler for vector [ 1560.452348] do_IRQ: 0.41 No irq handler for vector [ 1560.460293] do_IRQ: 0.41 No irq handler for vector [ 1560.465303] do_IRQ: 0.41 No irq handler for vector [ 1560.468033] do_IRQ: 0.41 No irq handler for vector [ 1568.215396] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 00:17:57 (1790741877) [ 1568.228735] do_IRQ: 1 callbacks suppressed [ 1568.228746] do_IRQ: 0.41 No irq handler for vector [ 1568.242187] do_IRQ: 0.41 No irq handler for vector [ 1568.247441] do_IRQ: 0.41 No irq handler for vector [ 1568.255679] do_IRQ: 0.41 No irq handler for vector [ 1568.262740] do_IRQ: 0.41 No irq handler for vector [ 1568.265714] do_IRQ: 0.41 No irq handler for vector [ 1568.270140] do_IRQ: 0.41 No irq handler for vector [ 1568.274296] do_IRQ: 0.41 No irq handler for vector [ 1568.279593] do_IRQ: 0.41 No irq handler for vector [ 1568.283856] do_IRQ: 0.41 No irq handler for vector [ 1568.895538] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.206.29@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3830 extent [0-4194303]: client csum 84d56f0f, server csum 84d56f0e [ 1571.851610] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1573.250570] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.206.29@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum c4eec430 (type 1), server csum f1e1570d (type 1) [ 1573.266862] do_IRQ: 3 callbacks suppressed [ 1573.266872] do_IRQ: 0.41 No irq handler for vector [ 1573.274871] do_IRQ: 0.41 No irq handler for vector [ 1573.276668] do_IRQ: 0.41 No irq handler for vector [ 1573.278476] do_IRQ: 0.41 No irq handler for vector [ 1573.280304] do_IRQ: 0.41 No irq handler for vector [ 1573.282797] do_IRQ: 0.41 No irq handler for vector [ 1573.287121] do_IRQ: 0.41 No irq handler for vector [ 1573.289029] do_IRQ: 0.41 No irq handler for vector [ 1573.291589] do_IRQ: 0.41 No irq handler for vector [ 1573.294159] do_IRQ: 0.41 No irq handler for vector [ 1575.517096] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1576.962640] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.206.29@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum 780f4025 (type 2), server csum 996340b7 (type 2) [ 1579.336780] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1579.349403] do_IRQ: 3 callbacks suppressed [ 1579.349433] do_IRQ: 0.41 No irq handler for vector [ 1579.354118] do_IRQ: 0.41 No irq handler for vector [ 1579.356779] do_IRQ: 0.41 No irq handler for vector [ 1579.363903] do_IRQ: 0.41 No irq handler for vector [ 1579.366892] do_IRQ: 0.41 No irq handler for vector [ 1579.369070] do_IRQ: 0.41 No irq handler for vector [ 1579.371501] do_IRQ: 0.41 No irq handler for vector [ 1579.378466] do_IRQ: 0.41 No irq handler for vector [ 1579.382555] do_IRQ: 0.41 No irq handler for vector [ 1579.385692] do_IRQ: 0.41 No irq handler for vector [ 1580.792663] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.206.29@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum 5db11d3c (type 4), server csum 84d56f0e (type 4) [ 1583.240631] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1584.841356] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.206.29@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum f960f5d (type 10), server csum 8daf0ecb (type 10) [ 1584.876095] do_IRQ: 3 callbacks suppressed [ 1584.876117] do_IRQ: 0.41 No irq handler for vector [ 1584.881813] do_IRQ: 0.41 No irq handler for vector [ 1584.889202] do_IRQ: 0.41 No irq handler for vector [ 1584.894675] do_IRQ: 0.41 No irq handler for vector [ 1584.900970] do_IRQ: 0.41 No irq handler for vector [ 1584.907860] do_IRQ: 0.41 No irq handler for vector [ 1584.913589] do_IRQ: 0.41 No irq handler for vector [ 1584.921320] do_IRQ: 0.41 No irq handler for vector [ 1584.925978] do_IRQ: 0.41 No irq handler for vector [ 1584.933059] do_IRQ: 0.41 No irq handler for vector [ 1587.694411] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1588.998641] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.206.29@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum a599ffd4 (type 20), server csum 15a1ff42 (type 20) [ 1591.384647] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1591.386835] do_IRQ: 4 callbacks suppressed [ 1591.386854] do_IRQ: 0.41 No irq handler for vector [ 1591.389275] do_IRQ: 0.41 No irq handler for vector [ 1591.393785] do_IRQ: 0.41 No irq handler for vector [ 1591.395962] do_IRQ: 0.41 No irq handler for vector [ 1591.400899] do_IRQ: 0.41 No irq handler for vector [ 1591.405167] do_IRQ: 0.41 No irq handler for vector [ 1591.406958] do_IRQ: 0.41 No irq handler for vector [ 1591.412506] do_IRQ: 0.41 No irq handler for vector [ 1591.422716] do_IRQ: 0.41 No irq handler for vector [ 1591.426325] do_IRQ: 0.41 No irq handler for vector [ 1595.478935] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1597.248920] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.206.29@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum 127ff705 (type 80), server csum d24ff6fd (type 80) [ 1597.266370] LustreError: Skipped 1 previous similar message [ 1597.266594] do_IRQ: 2 callbacks suppressed [ 1597.266599] do_IRQ: 0.41 No irq handler for vector [ 1597.273426] do_IRQ: 0.41 No irq handler for vector [ 1597.278535] do_IRQ: 0.41 No irq handler for vector [ 1597.282575] do_IRQ: 0.41 No irq handler for vector [ 1597.285306] do_IRQ: 0.41 No irq handler for vector [ 1597.288197] do_IRQ: 0.41 No irq handler for vector [ 1597.294280] do_IRQ: 0.41 No irq handler for vector [ 1597.298750] do_IRQ: 0.41 No irq handler for vector [ 1597.301326] do_IRQ: 0.41 No irq handler for vector [ 1597.304399] do_IRQ: 0.41 No irq handler for vector [ 1599.622511] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1607.799714] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 00:18:36 (1790741916) [ 1607.811187] do_IRQ: 2 callbacks suppressed [ 1607.811199] do_IRQ: 0.41 No irq handler for vector [ 1607.822414] do_IRQ: 0.41 No irq handler for vector [ 1607.826314] do_IRQ: 0.41 No irq handler for vector [ 1607.831342] do_IRQ: 0.41 No irq handler for vector [ 1607.835622] do_IRQ: 0.41 No irq handler for vector [ 1607.840393] do_IRQ: 0.41 No irq handler for vector [ 1607.848134] do_IRQ: 0.41 No irq handler for vector [ 1607.859843] do_IRQ: 0.41 No irq handler for vector [ 1607.861702] do_IRQ: 0.41 No irq handler for vector [ 1607.869093] do_IRQ: 0.41 No irq handler for vector [ 1615.928377] Lustre: 8436:0:(tgt_handler.c:2030:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000407:0xc20:0x0]:[0-1048575]-5b4a19ac-3118c975 [ 1615.957788] do_IRQ: 1 callbacks suppressed [ 1615.957798] do_IRQ: 0.41 No irq handler for vector [ 1615.978615] do_IRQ: 0.41 No irq handler for vector [ 1615.980124] do_IRQ: 0.41 No irq handler for vector [ 1615.999695] do_IRQ: 0.41 No irq handler for vector [ 1616.024450] do_IRQ: 0.41 No irq handler for vector [ 1616.026054] do_IRQ: 0.41 No irq handler for vector [ 1616.058882] do_IRQ: 0.41 No irq handler for vector [ 1616.060507] LustreError: dumping log to /tmp/lustre-log.1790741926.8436 [ 1616.069686] do_IRQ: 0.41 No irq handler for vector [ 1616.086836] do_IRQ: 0.41 No irq handler for vector [ 1616.107936] do_IRQ: 0.41 No irq handler for vector [ 1616.438488] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.206.29@tcp inode [0x200000407:0xc20:0x0] object 0x280000401:3840 extent [0-1048575], client returned csum 5b4a19ac (type 4), server csum 3118c975 (type 4) [ 1641.556413] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 00:19:10 (1790741950) [ 1641.566962] do_IRQ: 2 callbacks suppressed [ 1641.566974] do_IRQ: 0.41 No irq handler for vector [ 1641.570706] do_IRQ: 0.41 No irq handler for vector [ 1641.574662] do_IRQ: 0.41 No irq handler for vector [ 1641.578287] do_IRQ: 0.41 No irq handler for vector [ 1641.580635] do_IRQ: 0.41 No irq handler for vector [ 1641.586293] do_IRQ: 0.41 No irq handler for vector [ 1641.592303] do_IRQ: 0.41 No irq handler for vector [ 1641.597746] do_IRQ: 0.41 No irq handler for vector [ 1641.600567] do_IRQ: 0.41 No irq handler for vector [ 1641.604814] do_IRQ: 0.41 No irq handler for vector [ 1642.052963] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.206.29@tcp inode [0x200000407:0xc22:0x0] object 0x280000401:3841 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 1651.698421] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 00:19:20 (1790741960) [ 1651.709595] do_IRQ: 3 callbacks suppressed [ 1651.709608] do_IRQ: 0.41 No irq handler for vector [ 1651.723919] do_IRQ: 0.41 No irq handler for vector [ 1651.732442] do_IRQ: 0.41 No irq handler for vector [ 1651.734730] do_IRQ: 0.41 No irq handler for vector [ 1651.740593] do_IRQ: 0.41 No irq handler for vector [ 1651.749427] do_IRQ: 0.41 No irq handler for vector [ 1651.755133] do_IRQ: 0.41 No irq handler for vector [ 1651.758990] do_IRQ: 0.41 No irq handler for vector [ 1651.766739] do_IRQ: 0.41 No irq handler for vector [ 1651.775455] do_IRQ: 0.41 No irq handler for vector [ 1653.660223] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1654.029092] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.206.29@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3831 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 1657.404584] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1657.409812] do_IRQ: 3 callbacks suppressed [ 1657.409822] do_IRQ: 0.41 No irq handler for vector [ 1657.412054] do_IRQ: 0.41 No irq handler for vector [ 1657.416497] do_IRQ: 0.41 No irq handler for vector [ 1657.424330] do_IRQ: 0.41 No irq handler for vector [ 1657.429447] do_IRQ: 0.41 No irq handler for vector [ 1657.435429] do_IRQ: 0.41 No irq handler for vector [ 1657.440092] do_IRQ: 0.41 No irq handler for vector [ 1657.441856] do_IRQ: 0.41 No irq handler for vector [ 1657.449396] do_IRQ: 0.41 No irq handler for vector [ 1657.463782] do_IRQ: 0.41 No irq handler for vector [ 1657.987083] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.206.29@tcp inode [0x200000407:0xc24:0x0] object 0x280000401:3842 extent [0-4194303]: client csum 7d33b9a0, server csum 7d33b99f [ 1658.008411] LustreError: Skipped 3 previous similar messages [ 1661.098427] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1662.647223] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.206.29@tcp inode [0x200000407:0xc25:0x0] object 0x2c0000401:3832 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 1662.678078] do_IRQ: 5 callbacks suppressed [ 1662.678089] do_IRQ: 0.41 No irq handler for vector [ 1662.680960] do_IRQ: 0.41 No irq handler for vector [ 1662.684465] do_IRQ: 0.41 No irq handler for vector [ 1662.687228] do_IRQ: 0.41 No irq handler for vector [ 1662.693074] do_IRQ: 0.41 No irq handler for vector [ 1662.697155] do_IRQ: 0.41 No irq handler for vector [ 1662.700344] do_IRQ: 0.41 No irq handler for vector [ 1662.706347] do_IRQ: 0.41 No irq handler for vector [ 1662.711737] do_IRQ: 0.41 No irq handler for vector [ 1662.723387] do_IRQ: 0.41 No irq handler for vector [ 1662.734876] LustreError: Skipped 6 previous similar messages [ 1664.924137] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1668.371901] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1668.394013] do_IRQ: 4 callbacks suppressed [ 1668.394023] do_IRQ: 0.41 No irq handler for vector [ 1668.402125] do_IRQ: 0.41 No irq handler for vector [ 1668.411217] do_IRQ: 0.41 No irq handler for vector [ 1668.415143] do_IRQ: 0.41 No irq handler for vector [ 1668.419786] do_IRQ: 0.41 No irq handler for vector [ 1668.426505] do_IRQ: 0.41 No irq handler for vector [ 1668.430932] do_IRQ: 0.41 No irq handler for vector [ 1668.437375] do_IRQ: 0.41 No irq handler for vector [ 1668.443119] do_IRQ: 0.41 No irq handler for vector [ 1668.447125] do_IRQ: 0.41 No irq handler for vector [ 1671.748357] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1672.096866] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.206.29@tcp inode [0x200000407:0xc28:0x0] object 0x280000401:3844 extent [0-4194303]: client csum a30822f0, server csum a30822ef [ 1672.110162] LustreError: Skipped 8 previous similar messages [ 1674.972896] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1674.977290] do_IRQ: 4 callbacks suppressed [ 1674.977300] do_IRQ: 0.41 No irq handler for vector [ 1674.987946] do_IRQ: 0.41 No irq handler for vector [ 1674.992437] do_IRQ: 0.41 No irq handler for vector [ 1674.995224] do_IRQ: 0.41 No irq handler for vector [ 1674.997540] do_IRQ: 0.41 No irq handler for vector [ 1675.000083] do_IRQ: 0.41 No irq handler for vector [ 1675.003150] do_IRQ: 0.41 No irq handler for vector [ 1675.005381] do_IRQ: 0.41 No irq handler for vector [ 1675.009819] do_IRQ: 0.41 No irq handler for vector [ 1675.012626] do_IRQ: 0.41 No irq handler for vector [ 1683.860429] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1683.862690] do_IRQ: 1 callbacks suppressed [ 1683.862700] do_IRQ: 0.41 No irq handler for vector [ 1683.877231] do_IRQ: 0.41 No irq handler for vector [ 1683.879814] do_IRQ: 0.41 No irq handler for vector [ 1683.882145] do_IRQ: 0.41 No irq handler for vector [ 1683.884352] do_IRQ: 0.41 No irq handler for vector [ 1683.886850] do_IRQ: 0.41 No irq handler for vector [ 1683.889433] do_IRQ: 0.41 No irq handler for vector [ 1683.891667] do_IRQ: 0.41 No irq handler for vector [ 1683.894360] do_IRQ: 0.41 No irq handler for vector [ 1683.896803] do_IRQ: 0.41 No irq handler for vector [ 1685.450949] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 00:19:54 (1790741994) [ 1686.829216] Lustre: *** cfs_fail_loc=21a, val=0*** [ 1691.967748] Lustre: *** cfs_fail_loc=21b, val=0*** [ 1691.975917] do_IRQ: 3 callbacks suppressed [ 1691.975927] do_IRQ: 0.41 No irq handler for vector [ 1691.995797] do_IRQ: 0.41 No irq handler for vector [ 1692.004087] do_IRQ: 0.41 No irq handler for vector [ 1692.021794] do_IRQ: 0.41 No irq handler for vector [ 1692.024244] do_IRQ: 0.41 No irq handler for vector [ 1692.031043] do_IRQ: 0.41 No irq handler for vector [ 1692.033718] do_IRQ: 0.41 No irq handler for vector [ 1692.037993] do_IRQ: 0.41 No irq handler for vector [ 1692.045738] do_IRQ: 0.41 No irq handler for vector [ 1692.047753] do_IRQ: 0.41 No irq handler for vector [ 1701.723181] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 00:20:10 (1790742010) [ 1701.730945] do_IRQ: 1 callbacks suppressed [ 1701.730955] do_IRQ: 0.41 No irq handler for vector [ 1701.754547] do_IRQ: 0.41 No irq handler for vector [ 1701.756974] do_IRQ: 0.41 No irq handler for vector [ 1701.759673] do_IRQ: 0.41 No irq handler for vector [ 1701.763871] do_IRQ: 0.41 No irq handler for vector [ 1701.767681] do_IRQ: 0.41 No irq handler for vector [ 1701.771665] do_IRQ: 0.41 No irq handler for vector [ 1701.773680] do_IRQ: 0.41 No irq handler for vector [ 1701.781805] do_IRQ: 0.41 No irq handler for vector [ 1701.783081] do_IRQ: 0.41 No irq handler for vector [ 1702.515942] Lustre: Setting parameter lustre.osc.lustre*.checksums=0 in log params [ 1709.088605] Lustre: Modifying parameter lustre.osc.lustre*.checksums=1 in log params [ 1709.092585] do_IRQ: 2 callbacks suppressed [ 1709.092595] do_IRQ: 0.41 No irq handler for vector [ 1709.097639] do_IRQ: 0.41 No irq handler for vector [ 1709.100791] do_IRQ: 0.41 No irq handler for vector [ 1709.103335] do_IRQ: 0.41 No irq handler for vector [ 1709.106044] do_IRQ: 0.41 No irq handler for vector [ 1709.108972] do_IRQ: 0.41 No irq handler for vector [ 1709.113149] do_IRQ: 0.41 No irq handler for vector [ 1709.116615] do_IRQ: 0.41 No irq handler for vector [ 1709.122321] do_IRQ: 0.41 No irq handler for vector [ 1709.125588] do_IRQ: 0.41 No irq handler for vector [ 1715.178920] Lustre: Disabling parameter lustre.osc.lustre*.checksums= in log params [ 1715.187765] do_IRQ: 1 callbacks suppressed [ 1715.187774] do_IRQ: 0.41 No irq handler for vector [ 1715.207685] do_IRQ: 0.41 No irq handler for vector [ 1715.211544] do_IRQ: 0.41 No irq handler for vector [ 1715.217295] do_IRQ: 0.41 No irq handler for vector [ 1715.220542] do_IRQ: 0.41 No irq handler for vector [ 1715.223102] do_IRQ: 0.41 No irq handler for vector [ 1715.225473] do_IRQ: 0.41 No irq handler for vector [ 1715.227827] do_IRQ: 0.41 No irq handler for vector [ 1715.230163] do_IRQ: 0.41 No irq handler for vector [ 1715.232746] do_IRQ: 0.41 No irq handler for vector [ 1728.115729] Lustre: Setting parameter lustre.osc.lustre*.checksums=0 in log params [ 1728.121333] do_IRQ: 1 callbacks suppressed [ 1728.121343] do_IRQ: 0.41 No irq handler for vector [ 1728.135600] do_IRQ: 0.41 No irq handler for vector [ 1728.138041] do_IRQ: 0.41 No irq handler for vector [ 1728.145026] do_IRQ: 0.41 No irq handler for vector [ 1728.153756] do_IRQ: 0.41 No irq handler for vector [ 1728.164051] do_IRQ: 0.41 No irq handler for vector [ 1728.171345] do_IRQ: 0.41 No irq handler for vector [ 1728.173425] do_IRQ: 0.41 No irq handler for vector [ 1728.174920] do_IRQ: 0.41 No irq handler for vector [ 1728.177642] do_IRQ: 0.41 No irq handler for vector [ 1731.149561] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 00:20:40 (1790742040) [ 1733.153835] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 1733.162730] do_IRQ: 2 callbacks suppressed [ 1733.162743] do_IRQ: 0.41 No irq handler for vector [ 1733.184936] do_IRQ: 0.41 No irq handler for vector [ 1733.188661] do_IRQ: 0.41 No irq handler for vector [ 1733.191976] do_IRQ: 0.41 No irq handler for vector [ 1733.202982] do_IRQ: 0.41 No irq handler for vector [ 1733.206324] do_IRQ: 0.41 No irq handler for vector [ 1733.212108] do_IRQ: 0.41 No irq handler for vector [ 1733.214867] do_IRQ: 0.41 No irq handler for vector [ 1733.218015] do_IRQ: 0.41 No irq handler for vector [ 1733.223774] do_IRQ: 0.41 No irq handler for vector [ 1734.992619] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1740.161352] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid 50 [ 1740.173972] do_IRQ: 2 callbacks suppressed [ 1740.174049] do_IRQ: 0.41 No irq handler for vector [ 1740.179481] do_IRQ: 0.41 No irq handler for vector [ 1740.181526] do_IRQ: 0.41 No irq handler for vector [ 1740.184955] do_IRQ: 0.41 No irq handler for vector [ 1740.188984] do_IRQ: 0.41 No irq handler for vector [ 1740.193330] do_IRQ: 0.41 No irq handler for vector [ 1740.195050] do_IRQ: 0.41 No irq handler for vector [ 1740.197111] do_IRQ: 0.41 No irq handler for vector [ 1740.199876] do_IRQ: 0.41 No irq handler for vector [ 1740.205931] do_IRQ: 0.41 No irq handler for vector [ 1741.565108] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid in IDLE state after 0 sec [ 1746.783250] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid 50 [ 1746.796927] do_IRQ: 3 callbacks suppressed [ 1746.796939] do_IRQ: 0.41 No irq handler for vector [ 1746.799848] do_IRQ: 0.41 No irq handler for vector [ 1746.802605] do_IRQ: 0.41 No irq handler for vector [ 1746.805401] do_IRQ: 0.41 No irq handler for vector [ 1746.808946] do_IRQ: 0.41 No irq handler for vector [ 1746.813983] do_IRQ: 0.41 No irq handler for vector [ 1746.818274] do_IRQ: 0.41 No irq handler for vector [ 1746.820841] do_IRQ: 0.41 No irq handler for vector [ 1746.823627] do_IRQ: 0.41 No irq handler for vector [ 1746.825881] do_IRQ: 0.41 No irq handler for vector [ 1748.193261] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid in FULL state after 0 sec [ 1750.394377] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1756.966228] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid 50 [ 1756.971194] do_IRQ: 4 callbacks suppressed [ 1756.971203] do_IRQ: 0.41 No irq handler for vector [ 1756.979429] do_IRQ: 0.41 No irq handler for vector [ 1756.983812] do_IRQ: 0.41 No irq handler for vector [ 1756.986809] do_IRQ: 0.41 No irq handler for vector [ 1756.988401] do_IRQ: 0.41 No irq handler for vector [ 1756.990726] do_IRQ: 0.41 No irq handler for vector [ 1756.996297] do_IRQ: 0.41 No irq handler for vector [ 1756.998351] do_IRQ: 0.41 No irq handler for vector [ 1757.006194] do_IRQ: 0.41 No irq handler for vector [ 1757.009540] do_IRQ: 0.41 No irq handler for vector [ 1765.653358] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid in IDLE state after 6 sec [ 1765.664363] do_IRQ: 1 callbacks suppressed [ 1765.664371] do_IRQ: 0.41 No irq handler for vector [ 1765.677774] do_IRQ: 0.41 No irq handler for vector [ 1765.684592] do_IRQ: 0.41 No irq handler for vector [ 1765.697319] do_IRQ: 0.41 No irq handler for vector [ 1765.723211] do_IRQ: 0.41 No irq handler for vector [ 1765.726412] do_IRQ: 0.41 No irq handler for vector [ 1765.731690] do_IRQ: 0.41 No irq handler for vector [ 1765.749785] do_IRQ: 0.41 No irq handler for vector [ 1765.762968] do_IRQ: 0.41 No irq handler for vector [ 1765.765742] do_IRQ: 0.41 No irq handler for vector [ 1771.715456] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid 50 [ 1771.721681] do_IRQ: 1 callbacks suppressed [ 1771.721692] do_IRQ: 0.41 No irq handler for vector [ 1771.731773] do_IRQ: 0.41 No irq handler for vector [ 1771.734424] do_IRQ: 0.41 No irq handler for vector [ 1771.736607] do_IRQ: 0.41 No irq handler for vector [ 1771.738684] do_IRQ: 0.41 No irq handler for vector [ 1771.741484] do_IRQ: 0.41 No irq handler for vector [ 1771.743548] do_IRQ: 0.41 No irq handler for vector [ 1771.745568] do_IRQ: 0.41 No irq handler for vector [ 1771.747608] do_IRQ: 0.41 No irq handler for vector [ 1771.750436] do_IRQ: 0.41 No irq handler for vector [ 1773.121369] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid in FULL state after 0 sec [ 1774.857996] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1780.929806] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid 50 [ 1780.942413] do_IRQ: 3 callbacks suppressed [ 1780.942425] do_IRQ: 0.41 No irq handler for vector [ 1780.946686] do_IRQ: 0.41 No irq handler for vector [ 1780.949835] do_IRQ: 0.41 No irq handler for vector [ 1780.955794] do_IRQ: 0.41 No irq handler for vector [ 1780.961776] do_IRQ: 0.41 No irq handler for vector [ 1780.970293] do_IRQ: 0.41 No irq handler for vector [ 1780.973465] do_IRQ: 0.41 No irq handler for vector [ 1780.978590] do_IRQ: 0.41 No irq handler for vector [ 1780.982963] do_IRQ: 0.41 No irq handler for vector [ 1780.989994] do_IRQ: 0.41 No irq handler for vector [ 1791.114023] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid in IDLE state after 8 sec [ 1791.120527] do_IRQ: 2 callbacks suppressed [ 1791.120539] do_IRQ: 0.41 No irq handler for vector [ 1791.130659] do_IRQ: 0.41 No irq handler for vector [ 1791.132754] do_IRQ: 0.41 No irq handler for vector [ 1791.138167] do_IRQ: 0.41 No irq handler for vector [ 1791.140373] do_IRQ: 0.41 No irq handler for vector [ 1791.142511] do_IRQ: 0.41 No irq handler for vector [ 1791.144544] do_IRQ: 0.41 No irq handler for vector [ 1791.145989] do_IRQ: 0.41 No irq handler for vector [ 1791.148385] do_IRQ: 0.41 No irq handler for vector [ 1791.150618] do_IRQ: 0.41 No irq handler for vector [ 1796.538983] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid 50 [ 1796.546368] do_IRQ: 1 callbacks suppressed [ 1796.546378] do_IRQ: 0.41 No irq handler for vector [ 1796.554559] do_IRQ: 0.41 No irq handler for vector [ 1796.557642] do_IRQ: 0.41 No irq handler for vector [ 1796.561103] do_IRQ: 0.41 No irq handler for vector [ 1796.566276] do_IRQ: 0.41 No irq handler for vector [ 1796.570483] do_IRQ: 0.41 No irq handler for vector [ 1796.579469] do_IRQ: 0.41 No irq handler for vector [ 1796.583925] do_IRQ: 0.41 No irq handler for vector [ 1796.591401] do_IRQ: 0.41 No irq handler for vector [ 1796.597431] do_IRQ: 0.41 No irq handler for vector [ 1798.211662] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid in FULL state after 0 sec [ 1800.019390] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1806.146441] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid 50 [ 1806.161329] do_IRQ: 3 callbacks suppressed [ 1806.161339] do_IRQ: 0.41 No irq handler for vector [ 1806.177484] do_IRQ: 0.41 No irq handler for vector [ 1806.185332] do_IRQ: 0.41 No irq handler for vector [ 1806.191474] do_IRQ: 0.41 No irq handler for vector [ 1806.194460] do_IRQ: 0.41 No irq handler for vector [ 1806.202692] do_IRQ: 0.41 No irq handler for vector [ 1806.213573] do_IRQ: 0.41 No irq handler for vector [ 1806.215315] do_IRQ: 0.41 No irq handler for vector [ 1806.228531] do_IRQ: 0.41 No irq handler for vector [ 1806.235053] do_IRQ: 0.41 No irq handler for vector [ 1812.229638] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid in IDLE state after 4 sec [ 1812.236333] do_IRQ: 2 callbacks suppressed [ 1812.236345] do_IRQ: 0.41 No irq handler for vector [ 1812.246425] do_IRQ: 0.41 No irq handler for vector [ 1812.252173] do_IRQ: 0.41 No irq handler for vector [ 1812.265365] do_IRQ: 0.41 No irq handler for vector [ 1812.275557] do_IRQ: 0.41 No irq handler for vector [ 1812.282124] do_IRQ: 0.41 No irq handler for vector [ 1812.284863] do_IRQ: 0.41 No irq handler for vector [ 1812.292583] do_IRQ: 0.41 No irq handler for vector [ 1812.298197] do_IRQ: 0.41 No irq handler for vector [ 1812.302732] do_IRQ: 0.41 No irq handler for vector [ 1818.458273] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid 50 [ 1818.476746] do_IRQ: 1 callbacks suppressed [ 1818.476754] do_IRQ: 0.41 No irq handler for vector [ 1818.486131] do_IRQ: 0.41 No irq handler for vector [ 1818.492434] do_IRQ: 0.41 No irq handler for vector [ 1818.497792] do_IRQ: 0.41 No irq handler for vector [ 1818.505876] do_IRQ: 0.41 No irq handler for vector [ 1818.512747] do_IRQ: 0.41 No irq handler for vector [ 1818.515382] do_IRQ: 0.41 No irq handler for vector [ 1818.519885] do_IRQ: 0.41 No irq handler for vector [ 1818.525808] do_IRQ: 0.41 No irq handler for vector [ 1818.533692] do_IRQ: 0.41 No irq handler for vector [ 1820.343720] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid in FULL state after 0 sec [ 1821.952913] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1827.876343] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid 50 [ 1827.885804] do_IRQ: 3 callbacks suppressed [ 1827.885815] do_IRQ: 0.41 No irq handler for vector [ 1827.891116] do_IRQ: 0.41 No irq handler for vector [ 1827.903363] do_IRQ: 0.41 No irq handler for vector [ 1827.906340] do_IRQ: 0.41 No irq handler for vector [ 1827.910989] do_IRQ: 0.41 No irq handler for vector [ 1827.914774] do_IRQ: 0.41 No irq handler for vector [ 1827.922974] do_IRQ: 0.41 No irq handler for vector [ 1827.926500] do_IRQ: 0.41 No irq handler for vector [ 1827.934752] do_IRQ: 0.41 No irq handler for vector [ 1827.939971] do_IRQ: 0.41 No irq handler for vector [ 1836.763021] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid in IDLE state after 7 sec [ 1836.774823] do_IRQ: 1 callbacks suppressed [ 1836.774834] do_IRQ: 0.41 No irq handler for vector [ 1836.783540] do_IRQ: 0.41 No irq handler for vector [ 1836.787035] do_IRQ: 0.41 No irq handler for vector [ 1836.789974] do_IRQ: 0.41 No irq handler for vector [ 1836.806407] do_IRQ: 0.41 No irq handler for vector [ 1836.810546] do_IRQ: 0.41 No irq handler for vector [ 1836.813539] do_IRQ: 0.41 No irq handler for vector [ 1836.816532] do_IRQ: 0.41 No irq handler for vector [ 1836.819486] do_IRQ: 0.41 No irq handler for vector [ 1836.823205] do_IRQ: 0.41 No irq handler for vector [ 1842.448369] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid 50 [ 1842.467455] do_IRQ: 1 callbacks suppressed [ 1842.467463] do_IRQ: 0.41 No irq handler for vector [ 1842.474522] do_IRQ: 0.41 No irq handler for vector [ 1842.479726] do_IRQ: 0.41 No irq handler for vector [ 1842.486900] do_IRQ: 0.41 No irq handler for vector [ 1842.490060] do_IRQ: 0.41 No irq handler for vector [ 1842.492705] do_IRQ: 0.41 No irq handler for vector [ 1842.496146] do_IRQ: 0.41 No irq handler for vector [ 1842.502866] do_IRQ: 0.41 No irq handler for vector [ 1842.511957] do_IRQ: 0.41 No irq handler for vector [ 1842.517876] do_IRQ: 0.41 No irq handler for vector [ 1843.845336] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid in FULL state after 0 sec [ 1845.361846] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1850.934905] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid 50 [ 1850.943704] do_IRQ: 3 callbacks suppressed [ 1850.943713] do_IRQ: 0.41 No irq handler for vector [ 1850.947254] do_IRQ: 0.41 No irq handler for vector [ 1850.950996] do_IRQ: 0.41 No irq handler for vector [ 1850.955078] do_IRQ: 0.41 No irq handler for vector [ 1850.958275] do_IRQ: 0.41 No irq handler for vector [ 1850.961502] do_IRQ: 0.41 No irq handler for vector [ 1850.964370] do_IRQ: 0.41 No irq handler for vector [ 1850.969961] do_IRQ: 0.41 No irq handler for vector [ 1850.975195] do_IRQ: 0.41 No irq handler for vector [ 1850.980488] do_IRQ: 0.41 No irq handler for vector [ 1857.636699] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid in IDLE state after 5 sec [ 1857.645687] do_IRQ: 2 callbacks suppressed [ 1857.645699] do_IRQ: 0.41 No irq handler for vector [ 1857.651466] do_IRQ: 0.41 No irq handler for vector [ 1857.659932] do_IRQ: 0.41 No irq handler for vector [ 1857.664191] do_IRQ: 0.41 No irq handler for vector [ 1857.667941] do_IRQ: 0.41 No irq handler for vector [ 1857.672626] do_IRQ: 0.41 No irq handler for vector [ 1857.677494] do_IRQ: 0.41 No irq handler for vector [ 1857.683902] do_IRQ: 0.41 No irq handler for vector [ 1857.691339] do_IRQ: 0.41 No irq handler for vector [ 1857.695746] do_IRQ: 0.41 No irq handler for vector [ 1863.692407] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid 50 [ 1863.701166] do_IRQ: 1 callbacks suppressed [ 1863.701175] do_IRQ: 0.41 No irq handler for vector [ 1863.705421] do_IRQ: 0.41 No irq handler for vector [ 1863.708420] do_IRQ: 0.41 No irq handler for vector [ 1863.715330] do_IRQ: 0.41 No irq handler for vector [ 1863.725299] do_IRQ: 0.41 No irq handler for vector [ 1863.729246] do_IRQ: 0.41 No irq handler for vector [ 1863.734482] do_IRQ: 0.41 No irq handler for vector [ 1863.745330] do_IRQ: 0.41 No irq handler for vector [ 1863.751085] do_IRQ: 0.41 No irq handler for vector [ 1863.755882] do_IRQ: 0.41 No irq handler for vector [ 1865.209380] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid in FULL state after 0 sec [ 1866.795582] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1872.011578] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid 50 [ 1872.024364] do_IRQ: 4 callbacks suppressed [ 1872.024377] do_IRQ: 0.41 No irq handler for vector [ 1872.027463] do_IRQ: 0.41 No irq handler for vector [ 1872.031554] do_IRQ: 0.41 No irq handler for vector [ 1872.033599] do_IRQ: 0.41 No irq handler for vector [ 1872.035828] do_IRQ: 0.41 No irq handler for vector [ 1872.038020] do_IRQ: 0.41 No irq handler for vector [ 1872.040221] do_IRQ: 0.41 No irq handler for vector [ 1872.042270] do_IRQ: 0.41 No irq handler for vector [ 1872.044449] do_IRQ: 0.41 No irq handler for vector [ 1872.046012] do_IRQ: 0.41 No irq handler for vector [ 1883.662642] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid in IDLE state after 9 sec [ 1883.679311] do_IRQ: 2 callbacks suppressed [ 1883.679318] do_IRQ: 0.41 No irq handler for vector [ 1883.688489] do_IRQ: 0.41 No irq handler for vector [ 1883.694133] do_IRQ: 0.41 No irq handler for vector [ 1883.698518] do_IRQ: 0.41 No irq handler for vector [ 1883.701560] do_IRQ: 0.41 No irq handler for vector [ 1883.706265] do_IRQ: 0.41 No irq handler for vector [ 1883.711763] do_IRQ: 0.41 No irq handler for vector [ 1883.720328] do_IRQ: 0.41 No irq handler for vector [ 1883.723373] do_IRQ: 0.41 No irq handler for vector [ 1883.727028] do_IRQ: 0.41 No irq handler for vector [ 1888.719154] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid 50 [ 1888.726109] do_IRQ: 1 callbacks suppressed [ 1888.726117] do_IRQ: 0.41 No irq handler for vector [ 1888.735362] do_IRQ: 0.41 No irq handler for vector [ 1888.739594] do_IRQ: 0.41 No irq handler for vector [ 1888.743143] do_IRQ: 0.41 No irq handler for vector [ 1888.745976] do_IRQ: 0.41 No irq handler for vector [ 1888.747670] do_IRQ: 0.41 No irq handler for vector [ 1888.750464] do_IRQ: 0.41 No irq handler for vector [ 1888.753409] do_IRQ: 0.41 No irq handler for vector [ 1888.756275] do_IRQ: 0.41 No irq handler for vector [ 1888.758515] do_IRQ: 0.41 No irq handler for vector [ 1890.204573] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f83cd2f800.ost_server_uuid in FULL state after 0 sec [ 1896.542617] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1896.547603] do_IRQ: 2 callbacks suppressed [ 1896.547614] do_IRQ: 0.41 No irq handler for vector [ 1896.554716] do_IRQ: 0.41 No irq handler for vector [ 1896.557239] do_IRQ: 0.41 No irq handler for vector [ 1896.561220] do_IRQ: 0.41 No irq handler for vector [ 1896.567821] do_IRQ: 0.41 No irq handler for vector [ 1896.571582] do_IRQ: 0.41 No irq handler for vector [ 1896.574992] do_IRQ: 0.41 No irq handler for vector [ 1896.577274] do_IRQ: 0.41 No irq handler for vector [ 1896.580327] do_IRQ: 0.41 No irq handler for vector [ 1896.582813] do_IRQ: 0.41 No irq handler for vector [ 1898.580688] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 00:23:27 (1790742207) [ 1905.299938] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 00:23:34 (1790742214) [ 1905.314415] do_IRQ: 2 callbacks suppressed [ 1905.314424] do_IRQ: 0.41 No irq handler for vector [ 1905.323171] do_IRQ: 0.41 No irq handler for vector [ 1905.329091] do_IRQ: 0.41 No irq handler for vector [ 1905.331907] do_IRQ: 0.41 No irq handler for vector [ 1905.333543] do_IRQ: 0.41 No irq handler for vector [ 1905.335704] do_IRQ: 0.41 No irq handler for vector [ 1905.337365] do_IRQ: 0.41 No irq handler for vector [ 1905.339540] do_IRQ: 0.41 No irq handler for vector [ 1905.342065] do_IRQ: 0.41 No irq handler for vector [ 1905.344283] do_IRQ: 0.41 No irq handler for vector [ 1908.070392] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1910.128674] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1911.803532] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1911.807233] do_IRQ: 3 callbacks suppressed [ 1911.807245] do_IRQ: 0.41 No irq handler for vector [ 1911.811679] do_IRQ: 0.41 No irq handler for vector [ 1911.815029] do_IRQ: 0.41 No irq handler for vector [ 1911.816848] do_IRQ: 0.41 No irq handler for vector [ 1911.822910] do_IRQ: 0.41 No irq handler for vector [ 1911.827771] do_IRQ: 0.41 No irq handler for vector [ 1911.839354] do_IRQ: 0.41 No irq handler for vector [ 1911.843988] do_IRQ: 0.41 No irq handler for vector [ 1911.847052] do_IRQ: 0.41 No irq handler for vector [ 1911.850719] do_IRQ: 0.41 No irq handler for vector [ 1913.923776] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1921.884921] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1921.888177] do_IRQ: 2 callbacks suppressed [ 1921.888188] do_IRQ: 0.41 No irq handler for vector [ 1921.890594] do_IRQ: 0.41 No irq handler for vector [ 1921.897452] do_IRQ: 0.41 No irq handler for vector [ 1921.903088] do_IRQ: 0.41 No irq handler for vector [ 1921.910074] do_IRQ: 0.41 No irq handler for vector [ 1921.912640] do_IRQ: 0.41 No irq handler for vector [ 1921.923350] do_IRQ: 0.41 No irq handler for vector [ 1921.925129] do_IRQ: 0.41 No irq handler for vector [ 1921.927114] do_IRQ: 0.41 No irq handler for vector [ 1921.929009] do_IRQ: 0.41 No irq handler for vector [ 1923.722983] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 00:23:52 (1790742232) [ 1937.187817] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 00:24:05 (1790742245) [ 1937.200977] do_IRQ: 2 callbacks suppressed [ 1937.200986] do_IRQ: 0.41 No irq handler for vector [ 1937.206635] do_IRQ: 0.41 No irq handler for vector [ 1937.223513] do_IRQ: 0.41 No irq handler for vector [ 1937.232647] do_IRQ: 0.41 No irq handler for vector [ 1937.242553] do_IRQ: 0.41 No irq handler for vector [ 1937.258737] do_IRQ: 0.41 No irq handler for vector [ 1937.260421] do_IRQ: 0.41 No irq handler for vector [ 1937.262219] do_IRQ: 0.41 No irq handler for vector [ 1937.264783] do_IRQ: 0.41 No irq handler for vector [ 1937.267686] do_IRQ: 0.41 No irq handler for vector [ 1946.234796] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check =========== 00:24:15 (1790742255) [ 1946.238512] do_IRQ: 1 callbacks suppressed [ 1946.238522] do_IRQ: 0.41 No irq handler for vector [ 1946.241277] do_IRQ: 0.41 No irq handler for vector [ 1946.247124] do_IRQ: 0.41 No irq handler for vector [ 1946.248589] do_IRQ: 0.41 No irq handler for vector [ 1946.251685] do_IRQ: 0.41 No irq handler for vector [ 1946.253419] do_IRQ: 0.41 No irq handler for vector [ 1946.255368] do_IRQ: 0.41 No irq handler for vector [ 1946.257297] do_IRQ: 0.41 No irq handler for vector [ 1946.259122] do_IRQ: 0.41 No irq handler for vector [ 1946.260924] do_IRQ: 0.41 No irq handler for vector [ 1965.030705] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 00:24:33 (1790742273) [ 1965.037443] do_IRQ: 1 callbacks suppressed [ 1965.037455] do_IRQ: 0.41 No irq handler for vector [ 1965.048514] do_IRQ: 0.41 No irq handler for vector [ 1965.059104] do_IRQ: 0.41 No irq handler for vector [ 1965.062889] do_IRQ: 0.41 No irq handler for vector [ 1965.066754] do_IRQ: 0.41 No irq handler for vector [ 1965.070562] do_IRQ: 0.41 No irq handler for vector [ 1965.077404] do_IRQ: 0.41 No irq handler for vector [ 1965.079752] do_IRQ: 0.41 No irq handler for vector [ 1965.086363] do_IRQ: 0.41 No irq handler for vector [ 1965.099631] do_IRQ: 0.41 No irq handler for vector [ 1973.444582] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 00:24:42 (1790742282) [ 1973.462561] do_IRQ: 1 callbacks suppressed [ 1973.462571] do_IRQ: 0.41 No irq handler for vector [ 1973.472708] do_IRQ: 0.41 No irq handler for vector [ 1973.478464] do_IRQ: 0.41 No irq handler for vector [ 1973.484080] do_IRQ: 0.41 No irq handler for vector [ 1973.487380] do_IRQ: 0.41 No irq handler for vector [ 1973.489449] do_IRQ: 0.41 No irq handler for vector [ 1973.492357] do_IRQ: 0.41 No irq handler for vector [ 1973.496314] do_IRQ: 0.41 No irq handler for vector [ 1973.504930] do_IRQ: 0.41 No irq handler for vector [ 1973.511360] do_IRQ: 0.41 No irq handler for vector [ 1974.929142] Lustre: *** cfs_fail_loc=228, val=0*** [ 1982.348763] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 00:24:51 (1790742291) [ 1982.361065] do_IRQ: 2 callbacks suppressed [ 1982.361073] do_IRQ: 0.41 No irq handler for vector [ 1982.368869] do_IRQ: 0.41 No irq handler for vector [ 1982.370674] do_IRQ: 0.41 No irq handler for vector [ 1982.373353] do_IRQ: 0.41 No irq handler for vector [ 1982.375475] do_IRQ: 0.41 No irq handler for vector [ 1982.378976] do_IRQ: 0.41 No irq handler for vector [ 1982.384103] do_IRQ: 0.41 No irq handler for vector [ 1982.392332] do_IRQ: 0.41 No irq handler for vector [ 1982.393876] do_IRQ: 0.41 No irq handler for vector [ 1982.399285] do_IRQ: 0.41 No irq handler for vector [ 1984.031902] Lustre: *** cfs_fail_loc=228, val=0*** [ 1990.717955] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 00:24:59 (1790742299) [ 1990.726682] do_IRQ: 2 callbacks suppressed [ 1990.726692] do_IRQ: 0.41 No irq handler for vector [ 1990.729884] do_IRQ: 0.41 No irq handler for vector [ 1990.738026] do_IRQ: 0.41 No irq handler for vector [ 1990.740575] do_IRQ: 0.41 No irq handler for vector [ 1990.743766] do_IRQ: 0.41 No irq handler for vector [ 1990.747549] do_IRQ: 0.41 No irq handler for vector [ 1990.752915] do_IRQ: 0.41 No irq handler for vector [ 1990.759992] do_IRQ: 0.41 No irq handler for vector [ 1990.766911] do_IRQ: 0.41 No irq handler for vector [ 1990.772418] do_IRQ: 0.41 No irq handler for vector [ 2012.185277] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ========================================================== 00:25:21 (1790742321) [ 2012.191657] do_IRQ: 2 callbacks suppressed [ 2012.191670] do_IRQ: 0.41 No irq handler for vector [ 2012.203969] do_IRQ: 0.41 No irq handler for vector [ 2012.205673] do_IRQ: 0.41 No irq handler for vector [ 2012.215618] do_IRQ: 0.41 No irq handler for vector [ 2012.222222] do_IRQ: 0.41 No irq handler for vector [ 2012.225209] do_IRQ: 0.41 No irq handler for vector [ 2012.227872] do_IRQ: 0.41 No irq handler for vector [ 2012.231707] do_IRQ: 0.41 No irq handler for vector [ 2012.235921] do_IRQ: 0.41 No irq handler for vector [ 2012.239646] do_IRQ: 0.41 No irq handler for vector [ 2019.077880] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 00:25:28 (1790742328) [ 2019.087420] do_IRQ: 1 callbacks suppressed [ 2019.087430] do_IRQ: 0.41 No irq handler for vector [ 2019.089379] do_IRQ: 0.41 No irq handler for vector [ 2019.091700] do_IRQ: 0.41 No irq handler for vector [ 2019.094448] do_IRQ: 0.41 No irq handler for vector [ 2019.096815] do_IRQ: 0.41 No irq handler for vector [ 2019.104546] do_IRQ: 0.41 No irq handler for vector [ 2019.108755] do_IRQ: 0.41 No irq handler for vector [ 2019.111358] do_IRQ: 0.41 No irq handler for vector [ 2019.113515] do_IRQ: 0.41 No irq handler for vector [ 2019.115799] do_IRQ: 0.41 No irq handler for vector [ 2221.726055] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 00:28:50 (1790742530) [ 2221.734912] do_IRQ: 2 callbacks suppressed [ 2221.734920] do_IRQ: 0.41 No irq handler for vector [ 2221.744336] do_IRQ: 0.41 No irq handler for vector [ 2221.748476] do_IRQ: 0.41 No irq handler for vector [ 2221.751671] do_IRQ: 0.41 No irq handler for vector [ 2221.754738] do_IRQ: 0.41 No irq handler for vector [ 2221.758076] do_IRQ: 0.41 No irq handler for vector [ 2221.763116] do_IRQ: 0.41 No irq handler for vector [ 2221.767747] do_IRQ: 0.41 No irq handler for vector [ 2221.770642] do_IRQ: 0.41 No irq handler for vector [ 2221.772374] do_IRQ: 0.41 No irq handler for vector [ 2250.123638] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 00:29:19 (1790742559) [ 2250.130035] do_IRQ: 1 callbacks suppressed [ 2250.130046] do_IRQ: 0.41 No irq handler for vector [ 2250.134902] do_IRQ: 0.41 No irq handler for vector [ 2250.137789] do_IRQ: 0.41 No irq handler for vector [ 2250.139642] do_IRQ: 0.41 No irq handler for vector [ 2250.142860] do_IRQ: 0.41 No irq handler for vector [ 2250.145581] do_IRQ: 0.41 No irq handler for vector [ 2250.148219] do_IRQ: 0.41 No irq handler for vector [ 2250.150604] do_IRQ: 0.41 No irq handler for vector [ 2250.152913] do_IRQ: 0.41 No irq handler for vector [ 2250.154878] do_IRQ: 0.41 No irq handler for vector [ 2326.821477] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 00:30:35 (1790742635) [ 2326.839990] do_IRQ: 1 callbacks suppressed [ 2326.840000] do_IRQ: 0.41 No irq handler for vector [ 2326.855391] do_IRQ: 0.41 No irq handler for vector [ 2326.860991] do_IRQ: 0.41 No irq handler for vector [ 2326.864200] do_IRQ: 0.41 No irq handler for vector [ 2326.867912] do_IRQ: 0.41 No irq handler for vector [ 2326.879095] do_IRQ: 0.41 No irq handler for vector [ 2326.885774] do_IRQ: 0.41 No irq handler for vector [ 2326.902038] do_IRQ: 0.41 No irq handler for vector [ 2326.913616] do_IRQ: 0.41 No irq handler for vector [ 2326.915243] do_IRQ: 0.41 No irq handler for vector [ 2579.566787] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 00:34:48 (1790742888) [ 2579.589148] do_IRQ: 1 callbacks suppressed [ 2579.589160] do_IRQ: 0.41 No irq handler for vector [ 2579.597589] do_IRQ: 0.41 No irq handler for vector [ 2579.604013] do_IRQ: 0.41 No irq handler for vector [ 2579.611697] do_IRQ: 0.41 No irq handler for vector [ 2579.613429] do_IRQ: 0.41 No irq handler for vector [ 2579.629891] do_IRQ: 0.41 No irq handler for vector [ 2579.637971] do_IRQ: 0.41 No irq handler for vector [ 2579.650200] do_IRQ: 0.41 No irq handler for vector [ 2579.659417] do_IRQ: 0.41 No irq handler for vector [ 2579.674361] do_IRQ: 0.41 No irq handler for vector [ 2664.754607] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 00:36:13 (1790742973) [ 2664.762486] do_IRQ: 1 callbacks suppressed [ 2664.762496] do_IRQ: 0.41 No irq handler for vector [ 2664.778351] do_IRQ: 0.41 No irq handler for vector [ 2664.780972] do_IRQ: 0.41 No irq handler for vector [ 2664.782558] do_IRQ: 0.41 No irq handler for vector [ 2664.792108] do_IRQ: 0.41 No irq handler for vector [ 2664.795605] do_IRQ: 0.41 No irq handler for vector [ 2664.800003] do_IRQ: 0.41 No irq handler for vector [ 2664.807265] do_IRQ: 0.41 No irq handler for vector [ 2664.810252] do_IRQ: 0.41 No irq handler for vector [ 2664.812052] do_IRQ: 0.41 No irq handler for vector [ 2674.960850] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 00:36:23 (1790742983) [ 2674.982852] do_IRQ: 1 callbacks suppressed [ 2674.982862] do_IRQ: 0.41 No irq handler for vector [ 2674.986120] do_IRQ: 0.41 No irq handler for vector [ 2675.005887] do_IRQ: 0.41 No irq handler for vector [ 2675.019056] do_IRQ: 0.41 No irq handler for vector [ 2675.031536] do_IRQ: 0.41 No irq handler for vector [ 2675.033300] do_IRQ: 0.41 No irq handler for vector [ 2675.040892] do_IRQ: 0.41 No irq handler for vector [ 2675.051414] do_IRQ: 0.41 No irq handler for vector [ 2675.058098] do_IRQ: 0.41 No irq handler for vector [ 2675.064291] do_IRQ: 0.41 No irq handler for vector [ 2741.669334] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 00:37:30 (1790743050) [ 2741.677734] do_IRQ: 1 callbacks suppressed [ 2741.677745] do_IRQ: 0.41 No irq handler for vector [ 2741.681962] do_IRQ: 0.41 No irq handler for vector [ 2741.686479] do_IRQ: 0.41 No irq handler for vector [ 2741.688612] do_IRQ: 0.41 No irq handler for vector [ 2741.691268] do_IRQ: 0.41 No irq handler for vector [ 2741.693918] do_IRQ: 0.41 No irq handler for vector [ 2741.695701] do_IRQ: 0.41 No irq handler for vector [ 2741.699877] do_IRQ: 0.41 No irq handler for vector [ 2741.702441] do_IRQ: 0.41 No irq handler for vector [ 2741.704898] do_IRQ: 0.41 No irq handler for vector [ 2761.075695] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 00:37:49 (1790743069) [ 2761.092245] do_IRQ: 1 callbacks suppressed [ 2761.092259] do_IRQ: 0.41 No irq handler for vector [ 2761.103181] do_IRQ: 0.41 No irq handler for vector [ 2761.113514] do_IRQ: 0.41 No irq handler for vector [ 2761.126929] do_IRQ: 0.41 No irq handler for vector [ 2761.134552] do_IRQ: 0.41 No irq handler for vector [ 2761.140562] do_IRQ: 0.41 No irq handler for vector [ 2761.145803] do_IRQ: 0.41 No irq handler for vector [ 2761.155020] do_IRQ: 0.41 No irq handler for vector [ 2761.163829] do_IRQ: 0.41 No irq handler for vector [ 2761.172652] do_IRQ: 0.41 No irq handler for vector [ 2773.501653] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 00:38:02 (1790743082) [ 2773.510481] do_IRQ: 1 callbacks suppressed [ 2773.510491] do_IRQ: 0.41 No irq handler for vector [ 2773.512666] do_IRQ: 0.41 No irq handler for vector [ 2773.514613] do_IRQ: 0.41 No irq handler for vector [ 2773.516579] do_IRQ: 0.41 No irq handler for vector [ 2773.518283] do_IRQ: 0.41 No irq handler for vector [ 2773.520383] do_IRQ: 0.41 No irq handler for vector [ 2773.522461] do_IRQ: 0.41 No irq handler for vector [ 2773.525652] do_IRQ: 0.41 No irq handler for vector [ 2773.528696] do_IRQ: 0.41 No irq handler for vector [ 2773.532196] do_IRQ: 0.41 No irq handler for vector [ 2835.425211] Lustre: DEBUG MARKER: == sanity test 101m: read ahead for small file and last stripe of the file ========================================================== 00:39:04 (1790743144) [ 2835.440232] do_IRQ: 2 callbacks suppressed [ 2835.440241] do_IRQ: 0.41 No irq handler for vector [ 2835.448665] do_IRQ: 0.41 No irq handler for vector [ 2835.453704] do_IRQ: 0.41 No irq handler for vector [ 2835.456766] do_IRQ: 0.41 No irq handler for vector [ 2835.461930] do_IRQ: 0.41 No irq handler for vector [ 2835.467056] do_IRQ: 0.41 No irq handler for vector [ 2835.472761] do_IRQ: 0.41 No irq handler for vector [ 2835.477226] do_IRQ: 0.41 No irq handler for vector [ 2835.480591] do_IRQ: 0.41 No irq handler for vector [ 2835.483382] do_IRQ: 0.41 No irq handler for vector [ 2848.714438] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 00:39:17 (1790743157) [ 2848.722737] do_IRQ: 1 callbacks suppressed [ 2848.722746] do_IRQ: 0.41 No irq handler for vector [ 2848.731156] do_IRQ: 0.41 No irq handler for vector [ 2848.733578] do_IRQ: 0.41 No irq handler for vector [ 2848.739472] do_IRQ: 0.41 No irq handler for vector [ 2848.744778] do_IRQ: 0.41 No irq handler for vector [ 2848.746944] do_IRQ: 0.41 No irq handler for vector [ 2848.754646] do_IRQ: 0.41 No irq handler for vector [ 2848.761012] do_IRQ: 0.41 No irq handler for vector [ 2848.768732] do_IRQ: 0.41 No irq handler for vector [ 2848.777165] do_IRQ: 0.41 No irq handler for vector [ 2856.100958] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 00:39:25 (1790743165) [ 2856.112828] do_IRQ: 1 callbacks suppressed [ 2856.112838] do_IRQ: 0.41 No irq handler for vector [ 2856.119131] do_IRQ: 0.41 No irq handler for vector [ 2856.129586] do_IRQ: 0.41 No irq handler for vector [ 2856.138876] do_IRQ: 0.41 No irq handler for vector [ 2856.147845] do_IRQ: 0.41 No irq handler for vector [ 2856.153788] do_IRQ: 0.41 No irq handler for vector [ 2856.156295] do_IRQ: 0.41 No irq handler for vector [ 2856.160377] do_IRQ: 0.41 No irq handler for vector [ 2856.163342] do_IRQ: 0.41 No irq handler for vector [ 2856.168321] do_IRQ: 0.41 No irq handler for vector [ 2869.170835] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 00:39:38 (1790743178) [ 2869.182506] do_IRQ: 2 callbacks suppressed [ 2869.182517] do_IRQ: 0.41 No irq handler for vector [ 2869.198730] do_IRQ: 0.41 No irq handler for vector [ 2869.202840] do_IRQ: 0.41 No irq handler for vector [ 2869.207273] do_IRQ: 0.41 No irq handler for vector [ 2869.211215] do_IRQ: 0.41 No irq handler for vector [ 2869.214687] do_IRQ: 0.41 No irq handler for vector [ 2869.217421] do_IRQ: 0.41 No irq handler for vector [ 2869.223683] do_IRQ: 0.41 No irq handler for vector [ 2869.225490] do_IRQ: 0.41 No irq handler for vector [ 2869.229122] do_IRQ: 0.41 No irq handler for vector [ 2877.111435] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 00:39:45 (1790743185) [ 2877.134384] do_IRQ: 1 callbacks suppressed [ 2877.134394] do_IRQ: 0.41 No irq handler for vector [ 2877.141661] do_IRQ: 0.41 No irq handler for vector [ 2877.149111] do_IRQ: 0.41 No irq handler for vector [ 2877.158452] do_IRQ: 0.41 No irq handler for vector [ 2877.164808] do_IRQ: 0.41 No irq handler for vector [ 2877.170151] do_IRQ: 0.41 No irq handler for vector [ 2877.174738] do_IRQ: 0.41 No irq handler for vector [ 2877.184464] do_IRQ: 0.41 No irq handler for vector [ 2877.193154] do_IRQ: 0.41 No irq handler for vector [ 2877.199633] do_IRQ: 0.41 No irq handler for vector [ 2896.552831] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 00:40:05 (1790743205) [ 2896.566325] do_IRQ: 2 callbacks suppressed [ 2896.566336] do_IRQ: 0.41 No irq handler for vector [ 2896.571824] do_IRQ: 0.41 No irq handler for vector [ 2896.576549] do_IRQ: 0.41 No irq handler for vector [ 2896.583543] do_IRQ: 0.41 No irq handler for vector [ 2896.589108] do_IRQ: 0.41 No irq handler for vector [ 2896.592835] do_IRQ: 0.41 No irq handler for vector [ 2896.599596] do_IRQ: 0.41 No irq handler for vector [ 2896.609578] do_IRQ: 0.41 No irq handler for vector [ 2896.614572] do_IRQ: 0.41 No irq handler for vector [ 2896.616960] do_IRQ: 0.41 No irq handler for vector [ 2915.135483] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 00:40:24 (1790743224) [ 2915.145009] do_IRQ: 2 callbacks suppressed [ 2915.145019] do_IRQ: 0.41 No irq handler for vector [ 2915.163454] do_IRQ: 0.41 No irq handler for vector [ 2915.170823] do_IRQ: 0.41 No irq handler for vector [ 2915.180499] do_IRQ: 0.41 No irq handler for vector [ 2915.189150] do_IRQ: 0.41 No irq handler for vector [ 2915.195217] do_IRQ: 0.41 No irq handler for vector [ 2915.202986] do_IRQ: 0.41 No irq handler for vector [ 2915.208491] do_IRQ: 0.41 No irq handler for vector [ 2915.214817] do_IRQ: 0.41 No irq handler for vector [ 2915.217190] do_IRQ: 0.41 No irq handler for vector [ 2917.726596] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 2919.622708] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 2921.675379] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 2921.684947] do_IRQ: 3 callbacks suppressed [ 2921.684955] do_IRQ: 0.41 No irq handler for vector [ 2921.699688] do_IRQ: 0.41 No irq handler for vector [ 2921.708476] do_IRQ: 0.41 No irq handler for vector [ 2921.713498] do_IRQ: 0.41 No irq handler for vector [ 2921.727753] do_IRQ: 0.41 No irq handler for vector [ 2921.730302] do_IRQ: 0.41 No irq handler for vector [ 2921.736090] do_IRQ: 0.41 No irq handler for vector [ 2921.740198] do_IRQ: 0.41 No irq handler for vector [ 2921.741858] do_IRQ: 0.41 No irq handler for vector [ 2921.743523] do_IRQ: 0.41 No irq handler for vector [ 2923.479788] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2931.035229] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 00:40:40 (1790743240) [ 2931.045204] do_IRQ: 2 callbacks suppressed [ 2931.045214] do_IRQ: 0.41 No irq handler for vector [ 2931.057227] do_IRQ: 0.41 No irq handler for vector [ 2931.062699] do_IRQ: 0.41 No irq handler for vector [ 2931.068700] do_IRQ: 0.41 No irq handler for vector [ 2931.070423] do_IRQ: 0.41 No irq handler for vector [ 2931.079336] do_IRQ: 0.41 No irq handler for vector [ 2931.083519] do_IRQ: 0.41 No irq handler for vector [ 2931.086250] do_IRQ: 0.41 No irq handler for vector [ 2931.088846] do_IRQ: 0.41 No irq handler for vector [ 2931.090807] do_IRQ: 0.41 No irq handler for vector [ 2934.805649] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2937.226222] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 2937.230366] do_IRQ: 2 callbacks suppressed [ 2937.230376] do_IRQ: 0.41 No irq handler for vector [ 2937.236188] do_IRQ: 0.41 No irq handler for vector [ 2937.241798] do_IRQ: 0.41 No irq handler for vector [ 2937.244663] do_IRQ: 0.41 No irq handler for vector [ 2937.248193] do_IRQ: 0.41 No irq handler for vector [ 2937.252509] do_IRQ: 0.41 No irq handler for vector [ 2937.256107] do_IRQ: 0.41 No irq handler for vector [ 2937.259441] do_IRQ: 0.41 No irq handler for vector [ 2937.267434] do_IRQ: 0.41 No irq handler for vector [ 2937.270743] do_IRQ: 0.41 No irq handler for vector [ 2939.061670] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 2941.114414] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2944.190720] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2944.199125] do_IRQ: 4 callbacks suppressed [ 2944.199132] do_IRQ: 0.41 No irq handler for vector [ 2944.215725] do_IRQ: 0.41 No irq handler for vector [ 2944.218759] do_IRQ: 0.41 No irq handler for vector [ 2944.221018] do_IRQ: 0.41 No irq handler for vector [ 2944.225666] do_IRQ: 0.41 No irq handler for vector [ 2944.227838] do_IRQ: 0.41 No irq handler for vector [ 2944.233016] do_IRQ: 0.41 No irq handler for vector [ 2944.237267] do_IRQ: 0.41 No irq handler for vector [ 2944.242417] do_IRQ: 0.41 No irq handler for vector [ 2944.248277] do_IRQ: 0.41 No irq handler for vector [ 2951.442687] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 00:41:00 (1790743260) [ 2951.456330] do_IRQ: 1 callbacks suppressed [ 2951.456342] do_IRQ: 0.41 No irq handler for vector [ 2951.478217] do_IRQ: 0.41 No irq handler for vector [ 2951.484522] do_IRQ: 0.41 No irq handler for vector [ 2951.496284] do_IRQ: 0.41 No irq handler for vector [ 2951.500331] do_IRQ: 0.41 No irq handler for vector [ 2951.503302] do_IRQ: 0.41 No irq handler for vector [ 2951.506342] do_IRQ: 0.41 No irq handler for vector [ 2951.509470] do_IRQ: 0.41 No irq handler for vector [ 2951.516720] do_IRQ: 0.41 No irq handler for vector [ 2951.522594] do_IRQ: 0.41 No irq handler for vector [ 2959.446168] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 00:41:08 (1790743268) [ 2959.467041] do_IRQ: 1 callbacks suppressed [ 2959.467052] do_IRQ: 0.41 No irq handler for vector [ 2959.471467] do_IRQ: 0.41 No irq handler for vector [ 2959.480784] do_IRQ: 0.41 No irq handler for vector [ 2959.489396] do_IRQ: 0.41 No irq handler for vector [ 2959.497564] do_IRQ: 0.41 No irq handler for vector [ 2959.504711] do_IRQ: 0.41 No irq handler for vector [ 2959.520945] do_IRQ: 0.41 No irq handler for vector [ 2959.529597] do_IRQ: 0.41 No irq handler for vector [ 2959.541991] do_IRQ: 0.41 No irq handler for vector [ 2959.550340] do_IRQ: 0.41 No irq handler for vector [ 2979.012783] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 00:41:27 (1790743287) [ 2979.032863] do_IRQ: 1 callbacks suppressed [ 2979.032873] do_IRQ: 0.41 No irq handler for vector [ 2979.039949] do_IRQ: 0.41 No irq handler for vector [ 2979.047921] do_IRQ: 0.41 No irq handler for vector [ 2979.052206] do_IRQ: 0.41 No irq handler for vector [ 2979.054475] do_IRQ: 0.41 No irq handler for vector [ 2979.057562] do_IRQ: 0.41 No irq handler for vector [ 2979.063818] do_IRQ: 0.41 No irq handler for vector [ 2979.068802] do_IRQ: 0.41 No irq handler for vector [ 2979.073223] do_IRQ: 0.41 No irq handler for vector [ 2979.077062] do_IRQ: 0.41 No irq handler for vector [ 2987.676246] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 00:41:36 (1790743296) [ 2987.703742] do_IRQ: 2 callbacks suppressed [ 2987.703750] do_IRQ: 0.41 No irq handler for vector [ 2987.716354] do_IRQ: 0.41 No irq handler for vector [ 2987.721891] do_IRQ: 0.41 No irq handler for vector [ 2987.723613] do_IRQ: 0.41 No irq handler for vector [ 2987.735146] do_IRQ: 0.41 No irq handler for vector [ 2987.754931] do_IRQ: 0.41 No irq handler for vector [ 2987.762463] do_IRQ: 0.41 No irq handler for vector [ 2987.765832] do_IRQ: 0.41 No irq handler for vector [ 2987.770192] do_IRQ: 0.41 No irq handler for vector [ 2987.774229] do_IRQ: 0.41 No irq handler for vector [ 2995.363400] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 00:41:44 (1790743304) [ 2995.383032] do_IRQ: 1 callbacks suppressed [ 2995.383043] do_IRQ: 0.41 No irq handler for vector [ 2995.386761] do_IRQ: 0.41 No irq handler for vector [ 2995.402984] do_IRQ: 0.41 No irq handler for vector [ 2995.415200] do_IRQ: 0.41 No irq handler for vector [ 2995.423575] do_IRQ: 0.41 No irq handler for vector [ 2995.426063] do_IRQ: 0.41 No irq handler for vector [ 2995.428978] do_IRQ: 0.41 No irq handler for vector [ 2995.432026] do_IRQ: 0.41 No irq handler for vector [ 2995.434864] do_IRQ: 0.41 No irq handler for vector [ 2995.437349] do_IRQ: 0.41 No irq handler for vector [ 3002.807086] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 00:41:51 (1790743311) [ 3002.816743] do_IRQ: 1 callbacks suppressed [ 3002.816754] do_IRQ: 0.41 No irq handler for vector [ 3002.836474] do_IRQ: 0.41 No irq handler for vector [ 3002.839309] do_IRQ: 0.41 No irq handler for vector [ 3002.844492] do_IRQ: 0.41 No irq handler for vector [ 3002.847683] do_IRQ: 0.41 No irq handler for vector [ 3002.851139] do_IRQ: 0.41 No irq handler for vector [ 3002.859407] do_IRQ: 0.41 No irq handler for vector [ 3002.865672] do_IRQ: 0.41 No irq handler for vector [ 3002.879511] do_IRQ: 0.41 No irq handler for vector [ 3002.884198] do_IRQ: 0.41 No irq handler for vector [ 3011.379767] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 00:42:00 (1790743320) [ 3011.390229] do_IRQ: 1 callbacks suppressed [ 3011.390239] do_IRQ: 0.41 No irq handler for vector [ 3011.403026] do_IRQ: 0.41 No irq handler for vector [ 3011.409152] do_IRQ: 0.41 No irq handler for vector [ 3011.417142] do_IRQ: 0.41 No irq handler for vector [ 3011.421065] do_IRQ: 0.41 No irq handler for vector [ 3011.429460] do_IRQ: 0.41 No irq handler for vector [ 3011.435268] do_IRQ: 0.41 No irq handler for vector [ 3011.440739] do_IRQ: 0.41 No irq handler for vector [ 3011.447135] do_IRQ: 0.41 No irq handler for vector [ 3011.454067] do_IRQ: 0.41 No irq handler for vector [ 3019.114792] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 00:42:07 (1790743327) [ 3019.129660] do_IRQ: 1 callbacks suppressed [ 3019.129666] do_IRQ: 0.41 No irq handler for vector [ 3019.135655] do_IRQ: 0.41 No irq handler for vector [ 3019.140995] do_IRQ: 0.41 No irq handler for vector [ 3019.147497] do_IRQ: 0.41 No irq handler for vector [ 3019.160011] do_IRQ: 0.41 No irq handler for vector [ 3019.168905] do_IRQ: 0.41 No irq handler for vector [ 3019.170584] do_IRQ: 0.41 No irq handler for vector [ 3019.175990] do_IRQ: 0.41 No irq handler for vector [ 3019.180530] do_IRQ: 0.41 No irq handler for vector [ 3019.183190] do_IRQ: 0.41 No irq handler for vector [ 3026.491507] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 00:42:15 (1790743335) [ 3026.501602] do_IRQ: 1 callbacks suppressed [ 3026.501614] do_IRQ: 0.41 No irq handler for vector [ 3026.517668] do_IRQ: 0.41 No irq handler for vector [ 3026.522869] do_IRQ: 0.41 No irq handler for vector [ 3026.527335] do_IRQ: 0.41 No irq handler for vector [ 3026.534487] do_IRQ: 0.41 No irq handler for vector [ 3026.540622] do_IRQ: 0.41 No irq handler for vector [ 3026.546806] do_IRQ: 0.41 No irq handler for vector [ 3026.553394] do_IRQ: 0.41 No irq handler for vector [ 3026.557484] do_IRQ: 0.41 No irq handler for vector [ 3026.563238] do_IRQ: 0.41 No irq handler for vector [ 3034.440696] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 00:42:23 (1790743343) [ 3034.452010] do_IRQ: 1 callbacks suppressed [ 3034.452017] do_IRQ: 0.41 No irq handler for vector [ 3034.460376] do_IRQ: 0.41 No irq handler for vector [ 3034.463391] do_IRQ: 0.41 No irq handler for vector [ 3034.466291] do_IRQ: 0.41 No irq handler for vector [ 3034.468800] do_IRQ: 0.41 No irq handler for vector [ 3034.471701] do_IRQ: 0.41 No irq handler for vector [ 3034.474522] do_IRQ: 0.41 No irq handler for vector [ 3034.476883] do_IRQ: 0.41 No irq handler for vector [ 3034.480266] do_IRQ: 0.41 No irq handler for vector [ 3034.483247] do_IRQ: 0.41 No irq handler for vector [ 3041.669646] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 00:42:30 (1790743350) [ 3041.681310] do_IRQ: 1 callbacks suppressed [ 3041.681321] do_IRQ: 0.41 No irq handler for vector [ 3041.689478] do_IRQ: 0.41 No irq handler for vector [ 3041.694476] do_IRQ: 0.41 No irq handler for vector [ 3041.702835] do_IRQ: 0.41 No irq handler for vector [ 3041.708705] do_IRQ: 0.41 No irq handler for vector [ 3041.714426] do_IRQ: 0.41 No irq handler for vector [ 3041.718618] do_IRQ: 0.41 No irq handler for vector [ 3041.725486] do_IRQ: 0.41 No irq handler for vector [ 3041.731639] do_IRQ: 0.41 No irq handler for vector [ 3041.735570] do_IRQ: 0.41 No irq handler for vector [ 3049.327613] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 00:42:38 (1790743358) [ 3049.338262] do_IRQ: 2 callbacks suppressed [ 3049.338268] do_IRQ: 0.41 No irq handler for vector [ 3049.340950] do_IRQ: 0.41 No irq handler for vector [ 3049.347621] do_IRQ: 0.41 No irq handler for vector [ 3049.350345] do_IRQ: 0.41 No irq handler for vector [ 3049.352599] do_IRQ: 0.41 No irq handler for vector [ 3049.355278] do_IRQ: 0.41 No irq handler for vector [ 3049.357936] do_IRQ: 0.41 No irq handler for vector [ 3049.360646] do_IRQ: 0.41 No irq handler for vector [ 3049.363265] do_IRQ: 0.41 No irq handler for vector [ 3049.365579] do_IRQ: 0.41 No irq handler for vector [ 3310.808454] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 00:46:59 (1790743619) [ 3310.820922] do_IRQ: 1 callbacks suppressed [ 3310.820932] do_IRQ: 0.41 No irq handler for vector [ 3310.825956] do_IRQ: 0.41 No irq handler for vector [ 3310.829859] do_IRQ: 0.41 No irq handler for vector [ 3310.836595] do_IRQ: 0.41 No irq handler for vector [ 3310.843453] do_IRQ: 0.41 No irq handler for vector [ 3310.847970] do_IRQ: 0.41 No irq handler for vector [ 3310.852538] do_IRQ: 0.41 No irq handler for vector [ 3310.857706] do_IRQ: 0.41 No irq handler for vector [ 3310.862619] do_IRQ: 0.41 No irq handler for vector [ 3310.869617] do_IRQ: 0.41 No irq handler for vector [ 3473.669924] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 00:49:42 (1790743782) [ 3473.681342] do_IRQ: 2 callbacks suppressed [ 3473.681352] do_IRQ: 0.41 No irq handler for vector [ 3473.688328] do_IRQ: 0.41 No irq handler for vector [ 3473.691598] do_IRQ: 0.41 No irq handler for vector [ 3473.695349] do_IRQ: 0.41 No irq handler for vector [ 3473.701378] do_IRQ: 0.41 No irq handler for vector [ 3473.704934] do_IRQ: 0.41 No irq handler for vector [ 3473.707962] do_IRQ: 0.41 No irq handler for vector [ 3473.717581] do_IRQ: 0.41 No irq handler for vector [ 3473.721802] do_IRQ: 0.41 No irq handler for vector [ 3473.724837] do_IRQ: 0.41 No irq handler for vector [ 3479.873509] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 00:49:49 (1790743789) [ 3479.883317] do_IRQ: 1 callbacks suppressed [ 3479.883324] do_IRQ: 0.41 No irq handler for vector [ 3479.888482] do_IRQ: 0.41 No irq handler for vector [ 3479.891227] do_IRQ: 0.41 No irq handler for vector [ 3479.893603] do_IRQ: 0.41 No irq handler for vector [ 3479.896504] do_IRQ: 0.41 No irq handler for vector [ 3479.899096] do_IRQ: 0.41 No irq handler for vector [ 3479.901638] do_IRQ: 0.41 No irq handler for vector [ 3479.906685] do_IRQ: 0.41 No irq handler for vector [ 3479.910998] do_IRQ: 0.41 No irq handler for vector [ 3479.914422] do_IRQ: 0.41 No irq handler for vector [ 3499.596035] Lustre: 60988:0:(osd_handler.c:2198:osd_trans_start()) lustre-MDT0000: credits 30549 > trans_max 3200 [ 3499.602499] do_IRQ: 1 callbacks suppressed [ 3499.602510] do_IRQ: 0.41 No irq handler for vector [ 3499.602932] Lustre: 60988:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 3499.602940] Lustre: 60988:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28292/0 [ 3499.602945] Lustre: 60988:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 20/142/0, punch: 0/0/0, quota 0/0/0 [ 3499.602949] Lustre: 60988:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 3499.602955] Lustre: 60988:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 3499.602960] CPU: 2 PID: 60988 Comm: mdt00_007 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 3499.602964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3499.602967] Call Trace: [ 3499.603029] ? dump_stack+0xbb/0x10e [ 3499.607538] do_IRQ: 0.41 No irq handler for vector [ 3499.612892] do_IRQ: 0.41 No irq handler for vector [ 3499.618257] ? osd_trans_start+0x8d3/0x9d0 [osd_ldiskfs] [ 3499.618333] ? dt_trans_start+0x1c/0x70 [obdclass] [ 3499.618493] ? top_trans_start+0x4fd/0xd00 [ptlrpc] [ 3499.629999] do_IRQ: 0.41 No irq handler for vector [ 3499.639402] ? lod_declare_ref_add+0x1a/0x30 [lod] [ 3499.642787] do_IRQ: 0.41 No irq handler for vector [ 3499.667512] do_IRQ: 0.41 No irq handler for vector [ 3499.683035] ? dt_declare_ref_add+0x6c/0x200 [obdclass] [ 3499.689679] do_IRQ: 0.41 No irq handler for vector [ 3499.697931] do_IRQ: 0.41 No irq handler for vector [ 3499.703145] ? lod_trans_start+0x109/0x4c0 [lod] [ 3499.711888] do_IRQ: 0.41 No irq handler for vector [ 3499.718457] do_IRQ: 0.41 No irq handler for vector [ 3499.722928] ? mdd_declare_links_add+0x60/0x90 [mdd] [ 3499.722990] ? mdd_trans_start+0x18/0x30 [mdd] [ 3499.723030] ? mdd_unlink+0x7a4/0x13c0 [mdd] [ 3499.723069] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [ 3499.807915] ? mdt_reint_rec+0x139/0x2b0 [mdt] [ 3499.814411] ? mdt_reint_internal+0x693/0xdc0 [mdt] [ 3499.822011] ? mdt_reint+0x163/0x190 [mdt] [ 3499.823424] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 3499.832384] ? tgt_request_handle+0x575/0x1f70 [ptlrpc] [ 3499.836697] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 3499.845228] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 3499.853251] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [ 3499.865163] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [ 3499.873328] ? kthread+0x1d1/0x200 [ 3499.879252] ? set_kthread_struct+0x70/0x70 [ 3499.883358] ? ret_from_fork+0x1f/0x30 [ 3509.784271] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 00:50:17 (1790743817) [ 3509.817476] do_IRQ: 27 callbacks suppressed [ 3509.817484] do_IRQ: 0.41 No irq handler for vector [ 3509.821925] do_IRQ: 0.41 No irq handler for vector [ 3509.824663] do_IRQ: 0.41 No irq handler for vector [ 3509.828225] do_IRQ: 0.41 No irq handler for vector [ 3509.833351] do_IRQ: 0.41 No irq handler for vector [ 3509.838685] do_IRQ: 0.41 No irq handler for vector [ 3509.841277] do_IRQ: 0.41 No irq handler for vector [ 3509.844537] do_IRQ: 0.41 No irq handler for vector [ 3509.847426] do_IRQ: 0.41 No irq handler for vector [ 3509.849896] do_IRQ: 0.41 No irq handler for vector [ 3513.682452] Lustre: lustre-MDD0000: changelog on [ 3516.200215] Lustre: lustre-MDD0001: changelog on [ 3516.202620] do_IRQ: 2 callbacks suppressed [ 3516.202631] do_IRQ: 0.41 No irq handler for vector [ 3516.215615] do_IRQ: 0.41 No irq handler for vector [ 3516.218487] do_IRQ: 0.41 No irq handler for vector [ 3516.221730] do_IRQ: 0.41 No irq handler for vector [ 3516.224922] do_IRQ: 0.41 No irq handler for vector [ 3516.228145] do_IRQ: 0.41 No irq handler for vector [ 3516.230576] do_IRQ: 0.41 No irq handler for vector [ 3516.233955] do_IRQ: 0.41 No irq handler for vector [ 3516.237597] do_IRQ: 0.41 No irq handler for vector [ 3516.239953] do_IRQ: 0.41 No irq handler for vector [ 3522.654609] Lustre: lustre-MDD0001: changelog off [ 3522.657088] do_IRQ: 1 callbacks suppressed [ 3522.657097] do_IRQ: 0.41 No irq handler for vector [ 3522.661224] do_IRQ: 0.41 No irq handler for vector [ 3522.671327] do_IRQ: 0.41 No irq handler for vector [ 3522.680833] do_IRQ: 0.41 No irq handler for vector [ 3522.686846] do_IRQ: 0.41 No irq handler for vector [ 3522.690818] do_IRQ: 0.41 No irq handler for vector [ 3522.709636] do_IRQ: 0.41 No irq handler for vector [ 3522.715959] do_IRQ: 0.41 No irq handler for vector [ 3522.725576] do_IRQ: 0.41 No irq handler for vector [ 3522.738843] do_IRQ: 0.41 No irq handler for vector [ 3525.304429] Lustre: lustre-MDD0000: changelog off [ 3527.914165] Lustre: DEBUG MARKER: == sanity test 103g: no MDS_GETXATTR storm for inodes without ACL_ACCESS (LU-17238) ========================================================== 00:50:37 (1790743837) [ 3527.940487] do_IRQ: 2 callbacks suppressed [ 3527.940496] do_IRQ: 0.41 No irq handler for vector [ 3527.946797] do_IRQ: 0.41 No irq handler for vector [ 3527.951662] do_IRQ: 0.41 No irq handler for vector [ 3527.955907] do_IRQ: 0.41 No irq handler for vector [ 3527.963737] do_IRQ: 0.41 No irq handler for vector [ 3527.967824] do_IRQ: 0.41 No irq handler for vector [ 3527.971168] do_IRQ: 0.41 No irq handler for vector [ 3527.974277] do_IRQ: 0.41 No irq handler for vector [ 3527.976552] do_IRQ: 0.41 No irq handler for vector [ 3527.979384] do_IRQ: 0.41 No irq handler for vector [ 3535.787901] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 00:50:44 (1790743844) [ 3535.809164] do_IRQ: 1 callbacks suppressed [ 3535.809175] do_IRQ: 0.41 No irq handler for vector [ 3535.813675] do_IRQ: 0.41 No irq handler for vector [ 3535.817032] do_IRQ: 0.41 No irq handler for vector [ 3535.819615] do_IRQ: 0.41 No irq handler for vector [ 3535.820997] do_IRQ: 0.41 No irq handler for vector [ 3535.825957] do_IRQ: 0.41 No irq handler for vector [ 3535.830332] do_IRQ: 0.41 No irq handler for vector [ 3535.843506] do_IRQ: 0.41 No irq handler for vector [ 3535.850460] do_IRQ: 0.41 No irq handler for vector [ 3535.856452] do_IRQ: 0.41 No irq handler for vector [ 3536.421338] Lustre: lustre-OST0000: Client ee2cbb9d-82ea-42a0-ab9e-e174a7011aa1 (at 192.168.206.29@tcp) reconnecting [ 3541.895616] Lustre: DEBUG MARKER: oleg629-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff91f81035b800.ost_server_uuid 50 [ 3541.903359] do_IRQ: 2 callbacks suppressed [ 3541.903369] do_IRQ: 0.41 No irq handler for vector [ 3541.909450] do_IRQ: 0.41 No irq handler for vector [ 3541.915755] do_IRQ: 0.41 No irq handler for vector [ 3541.924384] do_IRQ: 0.41 No irq handler for vector [ 3541.927620] do_IRQ: 0.41 No irq handler for vector [ 3541.930737] do_IRQ: 0.41 No irq handler for vector [ 3541.936202] do_IRQ: 0.41 No irq handler for vector [ 3541.942550] do_IRQ: 0.41 No irq handler for vector [ 3541.948025] do_IRQ: 0.41 No irq handler for vector [ 3541.951605] do_IRQ: 0.41 No irq handler for vector [ 3543.733796] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91f81035b800.ost_server_uuid in FULL state after 0 sec [ 3551.563729] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 00:51:00 (1790743860) [ 3551.582368] do_IRQ: 3 callbacks suppressed [ 3551.582377] do_IRQ: 0.41 No irq handler for vector [ 3551.588508] do_IRQ: 0.41 No irq handler for vector [ 3551.596445] do_IRQ: 0.41 No irq handler for vector [ 3551.601993] do_IRQ: 0.41 No irq handler for vector [ 3551.606451] do_IRQ: 0.41 No irq handler for vector [ 3551.609371] do_IRQ: 0.41 No irq handler for vector [ 3551.615948] do_IRQ: 0.41 No irq handler for vector [ 3551.630431] do_IRQ: 0.41 No irq handler for vector [ 3551.632743] do_IRQ: 0.41 No irq handler for vector [ 3551.637944] do_IRQ: 0.41 No irq handler for vector [ 3558.170985] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 00:51:07 (1790743867) [ 3558.200548] do_IRQ: 1 callbacks suppressed [ 3558.200559] do_IRQ: 0.41 No irq handler for vector [ 3558.207104] do_IRQ: 0.41 No irq handler for vector [ 3558.212957] do_IRQ: 0.41 No irq handler for vector [ 3558.224012] do_IRQ: 0.41 No irq handler for vector [ 3558.234220] do_IRQ: 0.41 No irq handler for vector [ 3558.237021] do_IRQ: 0.41 No irq handler for vector [ 3558.241348] do_IRQ: 0.41 No irq handler for vector [ 3558.249149] do_IRQ: 0.41 No irq handler for vector [ 3558.252774] do_IRQ: 0.41 No irq handler for vector [ 3558.256499] do_IRQ: 0.41 No irq handler for vector [ 3559.781811] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [ 3561.964992] Lustre: DEBUG MARKER: == sanity test 104d: runas -u 500 -g 500 lctl dl test ==== 00:51:10 (1790743870) [ 3568.331781] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 00:51:17 (1790743877) [ 3568.365775] do_IRQ: 4 callbacks suppressed [ 3568.365785] do_IRQ: 0.41 No irq handler for vector [ 3568.373077] do_IRQ: 0.41 No irq handler for vector [ 3568.386325] do_IRQ: 0.41 No irq handler for vector [ 3568.401105] do_IRQ: 0.41 No irq handler for vector [ 3568.406733] do_IRQ: 0.41 No irq handler for vector [ 3568.421334] do_IRQ: 0.41 No irq handler for vector [ 3568.429665] do_IRQ: 0.41 No irq handler for vector [ 3568.441743] do_IRQ: 0.41 No irq handler for vector [ 3568.450269] do_IRQ: 0.41 No irq handler for vector [ 3568.457946] do_IRQ: 0.41 No irq handler for vector [ 3574.993709] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 00:51:24 (1790743884) [ 3575.002233] do_IRQ: 1 callbacks suppressed [ 3575.002244] do_IRQ: 0.41 No irq handler for vector [ 3575.004531] do_IRQ: 0.41 No irq handler for vector [ 3575.009084] do_IRQ: 0.41 No irq handler for vector [ 3575.010877] do_IRQ: 0.41 No irq handler for vector [ 3575.014156] do_IRQ: 0.41 No irq handler for vector [ 3575.021900] do_IRQ: 0.41 No irq handler for vector [ 3575.024141] do_IRQ: 0.41 No irq handler for vector [ 3575.027674] do_IRQ: 0.41 No irq handler for vector [ 3575.029679] do_IRQ: 0.41 No irq handler for vector [ 3575.033084] do_IRQ: 0.41 No irq handler for vector [ 3582.332600] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 00:51:31 (1790743891) [ 3582.339766] do_IRQ: 1 callbacks suppressed [ 3582.339774] do_IRQ: 0.41 No irq handler for vector [ 3582.345708] do_IRQ: 0.41 No irq handler for vector [ 3582.347366] do_IRQ: 0.41 No irq handler for vector [ 3582.349490] do_IRQ: 0.41 No irq handler for vector [ 3582.351164] do_IRQ: 0.41 No irq handler for vector [ 3582.353016] do_IRQ: 0.41 No irq handler for vector [ 3582.354733] do_IRQ: 0.41 No irq handler for vector [ 3582.356322] do_IRQ: 0.41 No irq handler for vector [ 3582.357898] do_IRQ: 0.41 No irq handler for vector [ 3582.359430] do_IRQ: 0.41 No irq handler for vector [ 3589.518897] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 00:51:38 (1790743898) [ 3589.533133] do_IRQ: 1 callbacks suppressed [ 3589.533143] do_IRQ: 0.41 No irq handler for vector [ 3589.536572] do_IRQ: 0.41 No irq handler for vector [ 3589.544964] do_IRQ: 0.41 No irq handler for vector [ 3589.552282] do_IRQ: 0.41 No irq handler for vector [ 3589.556676] do_IRQ: 0.41 No irq handler for vector [ 3589.562247] do_IRQ: 0.41 No irq handler for vector [ 3589.564533] do_IRQ: 0.41 No irq handler for vector [ 3589.570354] do_IRQ: 0.41 No irq handler for vector [ 3589.575344] do_IRQ: 0.41 No irq handler for vector [ 3589.579119] do_IRQ: 0.41 No irq handler for vector [ 3608.769650] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 00:51:57 (1790743917) [ 3608.779439] do_IRQ: 2 callbacks suppressed [ 3608.779450] do_IRQ: 0.41 No irq handler for vector [ 3608.797579] do_IRQ: 0.41 No irq handler for vector [ 3608.800478] do_IRQ: 0.41 No irq handler for vector [ 3608.803507] do_IRQ: 0.41 No irq handler for vector [ 3608.811612] do_IRQ: 0.41 No irq handler for vector [ 3608.819296] do_IRQ: 0.41 No irq handler for vector [ 3608.824111] do_IRQ: 0.41 No irq handler for vector [ 3608.832408] do_IRQ: 0.41 No irq handler for vector [ 3608.838639] do_IRQ: 0.41 No irq handler for vector [ 3608.843483] do_IRQ: 0.41 No irq handler for vector [ 3616.267167] Lustre: DEBUG MARKER: == sanity test 105f: Enqueue same range flocks =========== 00:52:05 (1790743925) [ 3616.276078] do_IRQ: 1 callbacks suppressed [ 3616.276085] do_IRQ: 0.41 No irq handler for vector [ 3616.284925] do_IRQ: 0.41 No irq handler for vector [ 3616.287340] do_IRQ: 0.41 No irq handler for vector [ 3616.290118] do_IRQ: 0.41 No irq handler for vector [ 3616.292984] do_IRQ: 0.41 No irq handler for vector [ 3616.298019] do_IRQ: 0.41 No irq handler for vector [ 3616.304775] do_IRQ: 0.41 No irq handler for vector [ 3616.309708] do_IRQ: 0.41 No irq handler for vector [ 3616.312896] do_IRQ: 0.41 No irq handler for vector [ 3616.316477] do_IRQ: 0.41 No irq handler for vector [ 3623.093123] Lustre: DEBUG MARKER: == sanity test 105g: ldlm_lock_debug stack test ========== 00:52:12 (1790743932) [ 3623.099188] do_IRQ: 1 callbacks suppressed [ 3623.099198] do_IRQ: 0.41 No irq handler for vector [ 3623.101356] do_IRQ: 0.41 No irq handler for vector [ 3623.103583] do_IRQ: 0.41 No irq handler for vector [ 3623.105412] do_IRQ: 0.41 No irq handler for vector [ 3623.106573] do_IRQ: 0.41 No irq handler for vector [ 3623.109558] do_IRQ: 0.41 No irq handler for vector [ 3623.115426] do_IRQ: 0.41 No irq handler for vector [ 3623.120252] do_IRQ: 0.41 No irq handler for vector [ 3623.122571] do_IRQ: 0.41 No irq handler for vector [ 3623.125591] do_IRQ: 0.41 No irq handler for vector [ 3631.753252] Lustre: DEBUG MARKER: == sanity test 105h: Flock functional verify ============= 00:52:20 (1790743940) [ 3631.764991] do_IRQ: 2 callbacks suppressed [ 3631.765005] do_IRQ: 0.41 No irq handler for vector [ 3631.773280] do_IRQ: 0.41 No irq handler for vector [ 3631.777986] do_IRQ: 0.41 No irq handler for vector [ 3631.780960] do_IRQ: 0.41 No irq handler for vector [ 3631.788140] do_IRQ: 0.41 No irq handler for vector [ 3631.795116] do_IRQ: 0.41 No irq handler for vector [ 3631.799380] do_IRQ: 0.41 No irq handler for vector [ 3631.802944] do_IRQ: 0.41 No irq handler for vector [ 3631.805869] do_IRQ: 0.41 No irq handler for vector [ 3631.809067] do_IRQ: 0.41 No irq handler for vector [ 3638.903861] Lustre: DEBUG MARKER: == sanity test 105i: Flock deadlock verify =============== 00:52:28 (1790743948) [ 3638.912573] do_IRQ: 1 callbacks suppressed [ 3638.912583] do_IRQ: 0.41 No irq handler for vector [ 3638.923934] do_IRQ: 0.41 No irq handler for vector [ 3638.928877] do_IRQ: 0.41 No irq handler for vector [ 3638.930662] do_IRQ: 0.41 No irq handler for vector [ 3638.937409] do_IRQ: 0.41 No irq handler for vector [ 3638.940452] do_IRQ: 0.41 No irq handler for vector [ 3638.946674] do_IRQ: 0.41 No irq handler for vector [ 3638.954723] do_IRQ: 0.41 No irq handler for vector [ 3638.960552] do_IRQ: 0.41 No irq handler for vector [ 3638.966883] do_IRQ: 0.41 No irq handler for vector [ 3649.469237] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 00:52:38 (1790743958) [ 3649.478598] do_IRQ: 1 callbacks suppressed [ 3649.478607] do_IRQ: 0.41 No irq handler for vector [ 3649.488018] do_IRQ: 0.41 No irq handler for vector [ 3649.490173] do_IRQ: 0.41 No irq handler for vector [ 3649.491459] do_IRQ: 0.41 No irq handler for vector [ 3649.493687] do_IRQ: 0.41 No irq handler for vector [ 3649.494852] do_IRQ: 0.41 No irq handler for vector [ 3649.501330] do_IRQ: 0.41 No irq handler for vector [ 3649.505358] do_IRQ: 0.41 No irq handler for vector [ 3649.509928] do_IRQ: 0.41 No irq handler for vector [ 3649.513244] do_IRQ: 0.41 No irq handler for vector [ 3654.069640] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 00:52:43 (1790743963) [ 3661.732187] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 00:52:51 (1790743971) [ 3661.752447] do_IRQ: 2 callbacks suppressed [ 3661.752459] do_IRQ: 0.41 No irq handler for vector [ 3661.763264] do_IRQ: 0.41 No irq handler for vector [ 3661.767504] do_IRQ: 0.41 No irq handler for vector [ 3661.776325] do_IRQ: 0.41 No irq handler for vector [ 3661.780313] do_IRQ: 0.41 No irq handler for vector [ 3661.784836] do_IRQ: 0.41 No irq handler for vector [ 3661.786965] do_IRQ: 0.41 No irq handler for vector [ 3661.788528] do_IRQ: 0.41 No irq handler for vector [ 3661.793579] do_IRQ: 0.41 No irq handler for vector [ 3661.795418] do_IRQ: 0.41 No irq handler for vector [ 3668.637557] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 00:52:58 (1790743978) [ 3668.647554] do_IRQ: 1 callbacks suppressed [ 3668.647563] do_IRQ: 0.41 No irq handler for vector [ 3668.659420] do_IRQ: 0.41 No irq handler for vector [ 3668.661695] do_IRQ: 0.41 No irq handler for vector [ 3668.664046] do_IRQ: 0.41 No irq handler for vector [ 3668.668053] do_IRQ: 0.41 No irq handler for vector [ 3668.669749] do_IRQ: 0.41 No irq handler for vector [ 3668.671843] do_IRQ: 0.41 No irq handler for vector [ 3668.673436] do_IRQ: 0.41 No irq handler for vector [ 3668.675163] do_IRQ: 0.41 No irq handler for vector [ 3668.676487] do_IRQ: 0.41 No irq handler for vector [ 3825.273899] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 00:55:34 (1790744134) [ 3825.283620] do_IRQ: 1 callbacks suppressed [ 3825.283627] do_IRQ: 0.41 No irq handler for vector [ 3825.286317] do_IRQ: 0.41 No irq handler for vector [ 3825.292749] do_IRQ: 0.41 No irq handler for vector [ 3825.302862] do_IRQ: 0.41 No irq handler for vector [ 3825.307734] do_IRQ: 0.41 No irq handler for vector [ 3825.311503] do_IRQ: 0.41 No irq handler for vector [ 3825.315602] do_IRQ: 0.41 No irq handler for vector [ 3825.319453] do_IRQ: 0.41 No irq handler for vector [ 3825.320848] do_IRQ: 0.41 No irq handler for vector [ 3825.326400] do_IRQ: 0.41 No irq handler for vector [ 3828.239257] Lustre: *** cfs_fail_loc=147, val=0*** [ 3836.433700] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 00:55:45 (1790744145) [ 3836.444502] do_IRQ: 2 callbacks suppressed [ 3836.444510] do_IRQ: 0.41 No irq handler for vector [ 3836.457053] do_IRQ: 0.41 No irq handler for vector [ 3836.458813] do_IRQ: 0.41 No irq handler for vector [ 3836.464548] do_IRQ: 0.41 No irq handler for vector [ 3836.471583] do_IRQ: 0.41 No irq handler for vector [ 3836.474919] do_IRQ: 0.41 No irq handler for vector [ 3836.480724] do_IRQ: 0.41 No irq handler for vector [ 3836.483119] do_IRQ: 0.41 No irq handler for vector [ 3836.485608] do_IRQ: 0.41 No irq handler for vector [ 3836.490093] do_IRQ: 0.41 No irq handler for vector [ 3843.022239] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 00:55:52 (1790744152) [ 3843.031285] do_IRQ: 1 callbacks suppressed [ 3843.031296] do_IRQ: 0.41 No irq handler for vector [ 3843.039698] do_IRQ: 0.41 No irq handler for vector [ 3843.043375] do_IRQ: 0.41 No irq handler for vector [ 3843.047703] do_IRQ: 0.41 No irq handler for vector [ 3843.051763] do_IRQ: 0.41 No irq handler for vector [ 3843.054607] do_IRQ: 0.41 No irq handler for vector [ 3843.057387] do_IRQ: 0.41 No irq handler for vector [ 3843.063886] do_IRQ: 0.41 No irq handler for vector [ 3843.069324] do_IRQ: 0.41 No irq handler for vector [ 3843.073884] do_IRQ: 0.41 No irq handler for vector [ 3850.301818] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 00:55:59 (1790744159) [ 3850.312535] do_IRQ: 1 callbacks suppressed [ 3850.312580] do_IRQ: 0.41 No irq handler for vector [ 3850.322692] do_IRQ: 0.41 No irq handler for vector [ 3850.325826] do_IRQ: 0.41 No irq handler for vector [ 3850.331301] do_IRQ: 0.41 No irq handler for vector [ 3850.334771] do_IRQ: 0.41 No irq handler for vector [ 3850.339698] do_IRQ: 0.41 No irq handler for vector [ 3850.353204] do_IRQ: 0.41 No irq handler for vector [ 3850.364503] do_IRQ: 0.41 No irq handler for vector [ 3850.367756] do_IRQ: 0.41 No irq handler for vector [ 3850.377353] do_IRQ: 0.41 No irq handler for vector [ 3852.008937] Lustre: *** cfs_fail_loc=217, val=0*** [ 3852.013147] Lustre: Skipped 29 previous similar messages [ 3860.044216] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 3860.054924] do_IRQ: 3 callbacks suppressed [ 3860.054936] do_IRQ: 0.41 No irq handler for vector [ 3860.063522] do_IRQ: 0.41 No irq handler for vector [ 3860.091149] do_IRQ: 0.41 No irq handler for vector [ 3860.102696] do_IRQ: 0.41 No irq handler for vector [ 3860.110418] do_IRQ: 0.41 No irq handler for vector [ 3860.118483] do_IRQ: 0.41 No irq handler for vector [ 3860.120324] do_IRQ: 0.41 No irq handler for vector [ 3860.124625] do_IRQ: 0.41 No irq handler for vector [ 3860.134917] do_IRQ: 0.41 No irq handler for vector [ 3860.141051] do_IRQ: 0.41 No irq handler for vector [ 3862.054957] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 3863.792669] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 00:56:12 (1790744172) [ 3871.626774] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 00:56:20 (1790744180) [ 3871.638857] do_IRQ: 3 callbacks suppressed [ 3871.638867] do_IRQ: 0.41 No irq handler for vector [ 3871.646039] do_IRQ: 0.41 No irq handler for vector [ 3871.653269] do_IRQ: 0.41 No irq handler for vector [ 3871.660043] do_IRQ: 0.41 No irq handler for vector [ 3871.665779] do_IRQ: 0.41 No irq handler for vector [ 3871.670046] do_IRQ: 0.41 No irq handler for vector [ 3871.675595] do_IRQ: 0.41 No irq handler for vector [ 3871.685084] do_IRQ: 0.41 No irq handler for vector [ 3871.690492] do_IRQ: 0.41 No irq handler for vector [ 3871.692034] do_IRQ: 0.41 No irq handler for vector [ 3879.322487] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 00:56:28 (1790744188) [ 3879.339511] do_IRQ: 1 callbacks suppressed [ 3879.339525] do_IRQ: 0.41 No irq handler for vector [ 3879.344765] do_IRQ: 0.41 No irq handler for vector [ 3879.352018] do_IRQ: 0.41 No irq handler for vector [ 3879.360817] do_IRQ: 0.41 No irq handler for vector [ 3879.365195] do_IRQ: 0.41 No irq handler for vector [ 3879.369579] do_IRQ: 0.41 No irq handler for vector [ 3879.373023] do_IRQ: 0.41 No irq handler for vector [ 3879.376853] do_IRQ: 0.41 No irq handler for vector [ 3879.379818] do_IRQ: 0.41 No irq handler for vector [ 3879.388569] do_IRQ: 0.41 No irq handler for vector [ 3881.075558] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3882.172604] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3884.235499] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3887.284349] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3887.287510] do_IRQ: 5 callbacks suppressed [ 3887.287521] do_IRQ: 0.41 No irq handler for vector [ 3887.300931] do_IRQ: 0.41 No irq handler for vector [ 3887.304847] do_IRQ: 0.41 No irq handler for vector [ 3887.311000] do_IRQ: 0.41 No irq handler for vector [ 3887.315987] do_IRQ: 0.41 No irq handler for vector [ 3887.325595] do_IRQ: 0.41 No irq handler for vector [ 3887.334515] do_IRQ: 0.41 No irq handler for vector [ 3887.338101] do_IRQ: 0.41 No irq handler for vector [ 3887.340788] do_IRQ: 0.41 No irq handler for vector [ 3887.343829] do_IRQ: 0.41 No irq handler for vector [ 3891.394982] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3899.974307] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 00:56:49 (1790744209) [ 3899.981351] do_IRQ: 2 callbacks suppressed [ 3899.981361] do_IRQ: 0.41 No irq handler for vector [ 3899.985811] do_IRQ: 0.41 No irq handler for vector [ 3899.997544] do_IRQ: 0.41 No irq handler for vector [ 3900.003667] do_IRQ: 0.41 No irq handler for vector [ 3900.009625] do_IRQ: 0.41 No irq handler for vector [ 3900.020951] do_IRQ: 0.41 No irq handler for vector [ 3900.032713] do_IRQ: 0.41 No irq handler for vector [ 3900.040614] do_IRQ: 0.41 No irq handler for vector [ 3900.043912] do_IRQ: 0.41 No irq handler for vector [ 3900.047097] do_IRQ: 0.41 No irq handler for vector [ 3901.814248] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3915.164933] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 00:57:04 (1790744224) [ 3915.187773] do_IRQ: 3 callbacks suppressed [ 3915.187782] do_IRQ: 0.41 No irq handler for vector [ 3915.192622] do_IRQ: 0.41 No irq handler for vector [ 3915.196832] do_IRQ: 0.41 No irq handler for vector [ 3915.202838] do_IRQ: 0.41 No irq handler for vector [ 3915.215738] do_IRQ: 0.41 No irq handler for vector [ 3915.220090] do_IRQ: 0.41 No irq handler for vector [ 3915.223958] do_IRQ: 0.41 No irq handler for vector [ 3915.228605] do_IRQ: 0.41 No irq handler for vector [ 3915.233397] do_IRQ: 0.41 No irq handler for vector [ 3915.240653] do_IRQ: 0.41 No irq handler for vector [ 3924.928898] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 00:57:13 (1790744233) [ 3924.948751] do_IRQ: 1 callbacks suppressed [ 3924.948762] do_IRQ: 0.41 No irq handler for vector [ 3924.965889] do_IRQ: 0.41 No irq handler for vector [ 3924.971970] do_IRQ: 0.41 No irq handler for vector [ 3924.976899] do_IRQ: 0.41 No irq handler for vector [ 3924.980502] do_IRQ: 0.41 No irq handler for vector [ 3924.981736] do_IRQ: 0.41 No irq handler for vector [ 3924.990715] do_IRQ: 0.41 No irq handler for vector [ 3924.997292] do_IRQ: 0.41 No irq handler for vector [ 3924.998899] do_IRQ: 0.41 No irq handler for vector [ 3925.002561] do_IRQ: 0.41 No irq handler for vector [ 3927.425616] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3927.437818] Lustre: Skipped 3 previous similar messages [ 3948.635192] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 00:57:37 (1790744257) [ 3948.639957] do_IRQ: 3 callbacks suppressed [ 3948.639966] do_IRQ: 0.41 No irq handler for vector [ 3948.642593] do_IRQ: 0.41 No irq handler for vector [ 3948.645692] do_IRQ: 0.41 No irq handler for vector [ 3948.648381] do_IRQ: 0.41 No irq handler for vector [ 3948.650971] do_IRQ: 0.41 No irq handler for vector [ 3948.653530] do_IRQ: 0.41 No irq handler for vector [ 3948.656307] do_IRQ: 0.41 No irq handler for vector [ 3948.659834] do_IRQ: 0.41 No irq handler for vector [ 3948.663555] do_IRQ: 0.41 No irq handler for vector [ 3948.666290] do_IRQ: 0.41 No irq handler for vector [ 3955.472282] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 00:57:44 (1790744264) [ 3955.477440] do_IRQ: 2 callbacks suppressed [ 3955.477450] do_IRQ: 0.41 No irq handler for vector [ 3955.479467] do_IRQ: 0.41 No irq handler for vector [ 3955.484434] do_IRQ: 0.41 No irq handler for vector [ 3955.486978] do_IRQ: 0.41 No irq handler for vector [ 3955.489808] do_IRQ: 0.41 No irq handler for vector [ 3955.492451] do_IRQ: 0.41 No irq handler for vector [ 3955.500594] do_IRQ: 0.41 No irq handler for vector [ 3955.503449] do_IRQ: 0.41 No irq handler for vector [ 3955.506251] do_IRQ: 0.41 No irq handler for vector [ 3955.508592] do_IRQ: 0.41 No irq handler for vector [ 3962.210637] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 00:57:51 (1790744271) [ 3962.222118] do_IRQ: 2 callbacks suppressed [ 3962.222125] do_IRQ: 0.41 No irq handler for vector [ 3962.229366] do_IRQ: 0.41 No irq handler for vector [ 3962.232798] do_IRQ: 0.41 No irq handler for vector [ 3962.235457] do_IRQ: 0.41 No irq handler for vector [ 3962.237734] do_IRQ: 0.41 No irq handler for vector [ 3962.240325] do_IRQ: 0.41 No irq handler for vector [ 3962.248224] do_IRQ: 0.41 No irq handler for vector [ 3962.261375] do_IRQ: 0.41 No irq handler for vector [ 3962.268239] do_IRQ: 0.41 No irq handler for vector [ 3962.273956] do_IRQ: 0.41 No irq handler for vector [ 3973.333885] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 00:58:02 (1790744282) [ 3973.344784] do_IRQ: 1 callbacks suppressed [ 3973.344792] do_IRQ: 0.41 No irq handler for vector [ 3973.350051] do_IRQ: 0.41 No irq handler for vector [ 3973.352581] do_IRQ: 0.41 No irq handler for vector [ 3973.355345] do_IRQ: 0.41 No irq handler for vector [ 3973.359626] do_IRQ: 0.41 No irq handler for vector [ 3973.367331] do_IRQ: 0.41 No irq handler for vector [ 3973.369122] do_IRQ: 0.41 No irq handler for vector [ 3973.379474] do_IRQ: 0.41 No irq handler for vector [ 3973.384755] do_IRQ: 0.41 No irq handler for vector [ 3973.389952] do_IRQ: 0.41 No irq handler for vector [ 3980.097573] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 00:58:09 (1790744289) [ 3980.105479] do_IRQ: 1 callbacks suppressed [ 3980.105492] do_IRQ: 0.41 No irq handler for vector [ 3980.112833] do_IRQ: 0.41 No irq handler for vector [ 3980.117237] do_IRQ: 0.41 No irq handler for vector [ 3980.120925] do_IRQ: 0.41 No irq handler for vector [ 3980.123544] do_IRQ: 0.41 No irq handler for vector [ 3980.126357] do_IRQ: 0.41 No irq handler for vector [ 3980.128711] do_IRQ: 0.41 No irq handler for vector [ 3980.133382] do_IRQ: 0.41 No irq handler for vector [ 3980.138939] do_IRQ: 0.41 No irq handler for vector [ 3980.146529] do_IRQ: 0.41 No irq handler for vector [ 3987.185915] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 00:58:16 (1790744296) [ 3987.197355] do_IRQ: 1 callbacks suppressed [ 3987.197361] do_IRQ: 0.41 No irq handler for vector [ 3987.203516] do_IRQ: 0.41 No irq handler for vector [ 3987.206623] do_IRQ: 0.41 No irq handler for vector [ 3987.209637] do_IRQ: 0.41 No irq handler for vector [ 3987.211894] do_IRQ: 0.41 No irq handler for vector [ 3987.213626] do_IRQ: 0.41 No irq handler for vector [ 3987.221340] do_IRQ: 0.41 No irq handler for vector [ 3987.224107] do_IRQ: 0.41 No irq handler for vector [ 3987.227925] do_IRQ: 0.41 No irq handler for vector [ 3987.235445] do_IRQ: 0.41 No irq handler for vector [ 3993.395133] Lustre: DEBUG MARKER: == sanity test 119e: Basic tests of dio read and write at various sizes ========================================================== 00:58:22 (1790744302) [ 3993.406536] do_IRQ: 1 callbacks suppressed [ 3993.406543] do_IRQ: 0.41 No irq handler for vector [ 3993.414618] do_IRQ: 0.41 No irq handler for vector [ 3993.418639] do_IRQ: 0.41 No irq handler for vector [ 3993.427258] do_IRQ: 0.41 No irq handler for vector [ 3993.436895] do_IRQ: 0.41 No irq handler for vector [ 3993.441312] do_IRQ: 0.41 No irq handler for vector [ 3993.448406] do_IRQ: 0.41 No irq handler for vector [ 3993.452044] do_IRQ: 0.41 No irq handler for vector [ 3993.455915] do_IRQ: 0.41 No irq handler for vector [ 3993.459225] do_IRQ: 0.41 No irq handler for vector [ 4031.298608] Lustre: DEBUG MARKER: == sanity test 119f: dio vs dio race ===================== 00:59:00 (1790744340) [ 4031.303424] do_IRQ: 1 callbacks suppressed [ 4031.303433] do_IRQ: 0.41 No irq handler for vector [ 4031.305882] do_IRQ: 0.41 No irq handler for vector [ 4031.309248] do_IRQ: 0.41 No irq handler for vector [ 4031.312099] do_IRQ: 0.41 No irq handler for vector [ 4031.314765] do_IRQ: 0.41 No irq handler for vector [ 4031.317628] do_IRQ: 0.41 No irq handler for vector [ 4031.320296] do_IRQ: 0.41 No irq handler for vector [ 4031.322934] do_IRQ: 0.41 No irq handler for vector [ 4031.325525] do_IRQ: 0.41 No irq handler for vector [ 4031.328821] do_IRQ: 0.41 No irq handler for vector [ 4074.803685] Lustre: DEBUG MARKER: == sanity test 119g: dio vs buffered I/O race ============ 00:59:43 (1790744383) [ 4074.808154] do_IRQ: 2 callbacks suppressed [ 4074.808178] do_IRQ: 0.41 No irq handler for vector [ 4074.813924] do_IRQ: 0.41 No irq handler for vector [ 4074.817977] do_IRQ: 0.41 No irq handler for vector [ 4074.820963] do_IRQ: 0.41 No irq handler for vector [ 4074.823829] do_IRQ: 0.41 No irq handler for vector [ 4074.826122] do_IRQ: 0.41 No irq handler for vector [ 4074.828635] do_IRQ: 0.41 No irq handler for vector [ 4074.833912] do_IRQ: 0.41 No irq handler for vector [ 4074.835831] do_IRQ: 0.41 No irq handler for vector [ 4074.837530] do_IRQ: 0.41 No irq handler for vector [ 4120.627477] Lustre: DEBUG MARKER: == sanity test 119h: basic tests of memory unaligned dio ========================================================== 01:00:29 (1790744429) [ 4120.636529] do_IRQ: 1 callbacks suppressed [ 4120.636540] do_IRQ: 0.41 No irq handler for vector [ 4120.639418] do_IRQ: 0.41 No irq handler for vector [ 4120.643452] do_IRQ: 0.41 No irq handler for vector [ 4120.646822] do_IRQ: 0.41 No irq handler for vector [ 4120.650614] do_IRQ: 0.41 No irq handler for vector [ 4120.653684] do_IRQ: 0.41 No irq handler for vector [ 4120.656532] do_IRQ: 0.41 No irq handler for vector [ 4120.659993] do_IRQ: 0.41 No irq handler for vector [ 4120.663643] do_IRQ: 0.41 No irq handler for vector [ 4120.666969] do_IRQ: 0.41 No irq handler for vector [ 4153.158165] Lustre: DEBUG MARKER: SKIP: sanity test_119i skipping ALWAYS excluded test 119i [ 4153.161908] do_IRQ: 2 callbacks suppressed [ 4153.161918] do_IRQ: 0.41 No irq handler for vector [ 4153.171741] do_IRQ: 0.41 No irq handler for vector [ 4153.174474] do_IRQ: 0.41 No irq handler for vector [ 4153.176688] do_IRQ: 0.41 No irq handler for vector [ 4153.180737] do_IRQ: 0.41 No irq handler for vector [ 4153.183436] do_IRQ: 0.41 No irq handler for vector [ 4153.185899] do_IRQ: 0.41 No irq handler for vector [ 4153.187791] do_IRQ: 0.41 No irq handler for vector [ 4153.189990] do_IRQ: 0.41 No irq handler for vector [ 4153.192185] do_IRQ: 0.41 No irq handler for vector [ 4155.148611] Lustre: DEBUG MARKER: == sanity test 119j: basic tests of hybrid IO switching == 01:01:04 (1790744464) [ 4163.666457] Lustre: DEBUG MARKER: == sanity test 119k: hybrid IO counting with stats and disabling ========================================================== 01:01:12 (1790744472) [ 4163.680547] do_IRQ: 2 callbacks suppressed [ 4163.680559] do_IRQ: 0.41 No irq handler for vector [ 4163.682636] do_IRQ: 0.41 No irq handler for vector [ 4163.685962] do_IRQ: 0.41 No irq handler for vector [ 4163.687790] do_IRQ: 0.41 No irq handler for vector [ 4163.690118] do_IRQ: 0.41 No irq handler for vector [ 4163.692431] do_IRQ: 0.41 No irq handler for vector [ 4163.694411] do_IRQ: 0.41 No irq handler for vector [ 4163.696088] do_IRQ: 0.41 No irq handler for vector [ 4163.697486] do_IRQ: 0.41 No irq handler for vector [ 4163.701303] do_IRQ: 0.41 No irq handler for vector [ 4181.223777] Lustre: DEBUG MARKER: == sanity test 119m: Test DIO readv/writev: exercise iter duplication ========================================================== 01:01:30 (1790744490) [ 4181.235677] do_IRQ: 2 callbacks suppressed [ 4181.235687] do_IRQ: 0.41 No irq handler for vector [ 4181.237223] do_IRQ: 0.41 No irq handler for vector [ 4181.239217] do_IRQ: 0.41 No irq handler for vector [ 4181.241488] do_IRQ: 0.41 No irq handler for vector [ 4181.243707] do_IRQ: 0.41 No irq handler for vector [ 4181.246330] do_IRQ: 0.41 No irq handler for vector [ 4181.248250] do_IRQ: 0.41 No irq handler for vector [ 4181.250174] do_IRQ: 0.41 No irq handler for vector [ 4181.253105] do_IRQ: 0.41 No irq handler for vector [ 4181.255439] do_IRQ: 0.41 No irq handler for vector [ 4187.208977] Lustre: DEBUG MARKER: == sanity test 119n: Test Unaligned DIO readv() and writev() with unpatched ZFS ========================================================== 01:01:36 (1790744496) [ 4187.215221] do_IRQ: 2 callbacks suppressed [ 4187.215233] do_IRQ: 0.41 No irq handler for vector [ 4187.218674] do_IRQ: 0.41 No irq handler for vector [ 4187.225977] do_IRQ: 0.41 No irq handler for vector [ 4187.233710] do_IRQ: 0.41 No irq handler for vector [ 4187.244388] do_IRQ: 0.41 No irq handler for vector [ 4187.247037] do_IRQ: 0.41 No irq handler for vector [ 4187.258758] do_IRQ: 0.41 No irq handler for vector [ 4187.266969] do_IRQ: 0.41 No irq handler for vector [ 4187.273341] do_IRQ: 0.41 No irq handler for vector [ 4187.276786] do_IRQ: 0.41 No irq handler for vector [ 4188.910837] Lustre: DEBUG MARKER: SKIP: sanity test_119n need ZFS server without unaligned_dio support [ 4190.774120] Lustre: DEBUG MARKER: == sanity test 119o: Test Unaligned DIO readv() and writev() with unpatched servers ========================================================== 01:01:39 (1790744499) [ 4192.485375] Lustre: DEBUG MARKER: SKIP: sanity test_119o need ldiskfs without unaligned_dio support. [ 4192.491161] do_IRQ: 3 callbacks suppressed [ 4192.491173] do_IRQ: 0.41 No irq handler for vector [ 4192.506185] do_IRQ: 0.41 No irq handler for vector [ 4192.510376] do_IRQ: 0.41 No irq handler for vector [ 4192.513569] do_IRQ: 0.41 No irq handler for vector [ 4192.517189] do_IRQ: 0.41 No irq handler for vector [ 4192.520613] do_IRQ: 0.41 No irq handler for vector [ 4192.523650] do_IRQ: 0.41 No irq handler for vector [ 4192.536501] do_IRQ: 0.41 No irq handler for vector [ 4192.542403] do_IRQ: 0.41 No irq handler for vector [ 4192.548727] do_IRQ: 0.41 No irq handler for vector [ 4194.543262] Lustre: DEBUG MARKER: == sanity test 119p: Test Unaligned DIO readv() and writev() with patched servers ========================================================== 01:01:43 (1790744503) [ 4201.771398] Lustre: DEBUG MARKER: == sanity test 119q: Test patchded Unaligned DIO readv() and writev() ========================================================== 01:01:50 (1790744510) [ 4201.792891] do_IRQ: 2 callbacks suppressed [ 4201.792902] do_IRQ: 0.41 No irq handler for vector [ 4201.804452] do_IRQ: 0.41 No irq handler for vector [ 4201.807292] do_IRQ: 0.41 No irq handler for vector [ 4201.811394] do_IRQ: 0.41 No irq handler for vector [ 4201.814100] do_IRQ: 0.41 No irq handler for vector [ 4201.816396] do_IRQ: 0.41 No irq handler for vector [ 4201.819204] do_IRQ: 0.41 No irq handler for vector [ 4201.821643] do_IRQ: 0.41 No irq handler for vector [ 4201.823486] do_IRQ: 0.41 No irq handler for vector [ 4201.825144] do_IRQ: 0.41 No irq handler for vector [ 4213.409973] Lustre: DEBUG MARKER: == sanity test 119r: Test error handling in unaligned DIO user copy ========================================================== 01:02:02 (1790744522) [ 4213.418979] do_IRQ: 1 callbacks suppressed [ 4213.418989] do_IRQ: 0.41 No irq handler for vector [ 4213.429549] do_IRQ: 0.41 No irq handler for vector [ 4213.431634] do_IRQ: 0.41 No irq handler for vector [ 4213.434176] do_IRQ: 0.41 No irq handler for vector [ 4213.436288] do_IRQ: 0.41 No irq handler for vector [ 4213.439427] do_IRQ: 0.41 No irq handler for vector [ 4213.441974] do_IRQ: 0.41 No irq handler for vector [ 4213.444229] do_IRQ: 0.41 No irq handler for vector [ 4213.446549] do_IRQ: 0.41 No irq handler for vector [ 4213.448962] do_IRQ: 0.41 No irq handler for vector [ 4220.528760] Lustre: DEBUG MARKER: == sanity test 119s: full-size unaligned DIO packs matching bulk MDs ========================================================== 01:02:09 (1790744529) [ 4220.543921] do_IRQ: 1 callbacks suppressed [ 4220.543931] do_IRQ: 0.41 No irq handler for vector [ 4220.551168] do_IRQ: 0.41 No irq handler for vector [ 4220.556191] do_IRQ: 0.41 No irq handler for vector [ 4220.563778] do_IRQ: 0.41 No irq handler for vector [ 4220.572366] do_IRQ: 0.41 No irq handler for vector [ 4220.578832] do_IRQ: 0.41 No irq handler for vector [ 4220.583454] do_IRQ: 0.41 No irq handler for vector [ 4220.586468] do_IRQ: 0.41 No irq handler for vector [ 4220.589019] do_IRQ: 0.41 No irq handler for vector [ 4220.591438] do_IRQ: 0.41 No irq handler for vector [ 4222.104933] Lustre: DEBUG MARKER: SKIP: sanity test_119s need client page size larger than the server's [ 4223.837514] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 01:02:12 (1790744532) [ 4233.816800] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 01:02:22 (1790744542) [ 4233.821533] do_IRQ: 4 callbacks suppressed [ 4233.821543] do_IRQ: 0.41 No irq handler for vector [ 4233.840916] do_IRQ: 0.41 No irq handler for vector [ 4233.851171] do_IRQ: 0.41 No irq handler for vector [ 4233.857430] do_IRQ: 0.41 No irq handler for vector [ 4233.862621] do_IRQ: 0.41 No irq handler for vector [ 4233.866314] do_IRQ: 0.41 No irq handler for vector [ 4233.871262] do_IRQ: 0.41 No irq handler for vector [ 4233.879184] do_IRQ: 0.41 No irq handler for vector [ 4233.882059] do_IRQ: 0.41 No irq handler for vector [ 4233.885947] do_IRQ: 0.41 No irq handler for vector [ 4243.806625] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 01:02:32 (1790744552) [ 4243.813093] do_IRQ: 1 callbacks suppressed [ 4243.813106] do_IRQ: 0.41 No irq handler for vector [ 4243.829029] do_IRQ: 0.41 No irq handler for vector [ 4243.836098] do_IRQ: 0.41 No irq handler for vector [ 4243.843710] do_IRQ: 0.41 No irq handler for vector [ 4243.850946] do_IRQ: 0.41 No irq handler for vector [ 4243.859504] do_IRQ: 0.41 No irq handler for vector [ 4243.862255] do_IRQ: 0.41 No irq handler for vector [ 4243.865289] do_IRQ: 0.41 No irq handler for vector [ 4243.867249] do_IRQ: 0.41 No irq handler for vector [ 4243.871999] do_IRQ: 0.41 No irq handler for vector [ 4255.338547] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 01:02:44 (1790744564) [ 4255.347935] do_IRQ: 1 callbacks suppressed [ 4255.347947] do_IRQ: 0.41 No irq handler for vector [ 4255.359532] do_IRQ: 0.41 No irq handler for vector [ 4255.363444] do_IRQ: 0.41 No irq handler for vector [ 4255.369294] do_IRQ: 0.41 No irq handler for vector [ 4255.377144] do_IRQ: 0.41 No irq handler for vector [ 4255.383516] do_IRQ: 0.41 No irq handler for vector [ 4255.389750] do_IRQ: 0.41 No irq handler for vector [ 4255.395786] do_IRQ: 0.41 No irq handler for vector [ 4255.398498] do_IRQ: 0.41 No irq handler for vector [ 4255.402866] do_IRQ: 0.41 No irq handler for vector [ 4266.442785] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 01:02:55 (1790744575) [ 4266.457650] do_IRQ: 1 callbacks suppressed [ 4266.457658] do_IRQ: 0.41 No irq handler for vector [ 4266.466911] do_IRQ: 0.41 No irq handler for vector [ 4266.470113] do_IRQ: 0.41 No irq handler for vector [ 4266.474088] do_IRQ: 0.41 No irq handler for vector [ 4266.484029] do_IRQ: 0.41 No irq handler for vector [ 4266.486550] do_IRQ: 0.41 No irq handler for vector [ 4266.500206] do_IRQ: 0.41 No irq handler for vector [ 4266.509477] do_IRQ: 0.41 No irq handler for vector [ 4266.520175] do_IRQ: 0.41 No irq handler for vector [ 4266.529387] do_IRQ: 0.41 No irq handler for vector [ 4284.386711] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 01:03:13 (1790744593) [ 4284.393723] do_IRQ: 1 callbacks suppressed [ 4284.393733] do_IRQ: 0.41 No irq handler for vector [ 4284.400622] do_IRQ: 0.41 No irq handler for vector [ 4284.414669] do_IRQ: 0.41 No irq handler for vector [ 4284.419578] do_IRQ: 0.41 No irq handler for vector [ 4284.421416] do_IRQ: 0.41 No irq handler for vector [ 4284.426081] do_IRQ: 0.41 No irq handler for vector [ 4284.428934] do_IRQ: 0.41 No irq handler for vector [ 4284.431820] do_IRQ: 0.41 No irq handler for vector [ 4284.436586] do_IRQ: 0.41 No irq handler for vector [ 4284.439989] do_IRQ: 0.41 No irq handler for vector [ 4301.742579] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 01:03:30 (1790744610) [ 4301.759363] do_IRQ: 1 callbacks suppressed [ 4301.759371] do_IRQ: 0.41 No irq handler for vector [ 4301.765999] do_IRQ: 0.41 No irq handler for vector [ 4301.776421] do_IRQ: 0.41 No irq handler for vector [ 4301.778148] do_IRQ: 0.41 No irq handler for vector [ 4301.783586] do_IRQ: 0.41 No irq handler for vector [ 4301.788506] do_IRQ: 0.41 No irq handler for vector [ 4301.795982] do_IRQ: 0.41 No irq handler for vector [ 4301.802529] do_IRQ: 0.41 No irq handler for vector [ 4301.809134] do_IRQ: 0.41 No irq handler for vector [ 4301.814139] do_IRQ: 0.41 No irq handler for vector [ 4685.767094] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 01:09:54 (1790744994) [ 4685.777230] do_IRQ: 1 callbacks suppressed [ 4685.777241] do_IRQ: 0.41 No irq handler for vector [ 4685.782567] do_IRQ: 0.41 No irq handler for vector [ 4685.788446] do_IRQ: 0.41 No irq handler for vector [ 4685.794738] do_IRQ: 0.41 No irq handler for vector [ 4685.802580] do_IRQ: 0.41 No irq handler for vector [ 4685.805530] do_IRQ: 0.41 No irq handler for vector [ 4685.807739] do_IRQ: 0.41 No irq handler for vector [ 4685.810569] do_IRQ: 0.41 No irq handler for vector [ 4685.812842] do_IRQ: 0.41 No irq handler for vector [ 4685.815813] do_IRQ: 0.41 No irq handler for vector [ 4692.225199] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 01:10:01 (1790745001) [ 4692.233405] do_IRQ: 2 callbacks suppressed [ 4692.233418] do_IRQ: 0.41 No irq handler for vector [ 4692.242154] do_IRQ: 0.41 No irq handler for vector [ 4692.246771] do_IRQ: 0.41 No irq handler for vector [ 4692.249820] do_IRQ: 0.41 No irq handler for vector [ 4692.252666] do_IRQ: 0.41 No irq handler for vector [ 4692.254826] do_IRQ: 0.41 No irq handler for vector [ 4692.256895] do_IRQ: 0.41 No irq handler for vector [ 4692.259462] do_IRQ: 0.41 No irq handler for vector [ 4692.261594] do_IRQ: 0.41 No irq handler for vector [ 4692.263819] do_IRQ: 0.41 No irq handler for vector [ 4701.001782] Lustre: DEBUG MARKER: 'ls -l' 100 files without statahead: 2 sec [ 4701.006978] do_IRQ: 1 callbacks suppressed [ 4701.006986] do_IRQ: 0.41 No irq handler for vector [ 4701.017946] do_IRQ: 0.41 No irq handler for vector [ 4701.020828] do_IRQ: 0.41 No irq handler for vector [ 4701.023506] do_IRQ: 0.41 No irq handler for vector [ 4701.027322] do_IRQ: 0.41 No irq handler for vector [ 4701.035112] do_IRQ: 0.41 No irq handler for vector [ 4701.042266] do_IRQ: 0.41 No irq handler for vector [ 4701.046593] do_IRQ: 0.41 No irq handler for vector [ 4701.057443] do_IRQ: 0.41 No irq handler for vector [ 4701.064292] do_IRQ: 0.41 No irq handler for vector [ 4703.660848] Lustre: DEBUG MARKER: 'ls -l' 100 files with statahead: 1 sec [ 4740.187793] Lustre: DEBUG MARKER: 'ls -l' 1000 files without statahead: 20 sec [ 4740.197631] do_IRQ: 2 callbacks suppressed [ 4740.197642] do_IRQ: 0.41 No irq handler for vector [ 4740.202371] do_IRQ: 0.41 No irq handler for vector [ 4740.209690] do_IRQ: 0.41 No irq handler for vector [ 4740.216013] do_IRQ: 0.41 No irq handler for vector [ 4740.220561] do_IRQ: 0.41 No irq handler for vector [ 4740.228078] do_IRQ: 0.41 No irq handler for vector [ 4740.231513] do_IRQ: 0.41 No irq handler for vector [ 4740.234374] do_IRQ: 0.41 No irq handler for vector [ 4740.237057] do_IRQ: 0.41 No irq handler for vector [ 4740.242865] do_IRQ: 0.41 No irq handler for vector [ 4747.241101] Lustre: DEBUG MARKER: 'ls -l' 1000 files with statahead: 5 sec [ 4747.249388] do_IRQ: 2 callbacks suppressed [ 4747.249397] do_IRQ: 0.41 No irq handler for vector [ 4747.258483] do_IRQ: 0.41 No irq handler for vector [ 4747.262682] do_IRQ: 0.41 No irq handler for vector [ 4747.267776] do_IRQ: 0.41 No irq handler for vector [ 4747.269675] do_IRQ: 0.41 No irq handler for vector [ 4747.273688] do_IRQ: 0.41 No irq handler for vector [ 4747.280151] do_IRQ: 0.41 No irq handler for vector [ 4747.283568] do_IRQ: 0.41 No irq handler for vector [ 4747.287500] do_IRQ: 0.41 No irq handler for vector [ 4747.294967] do_IRQ: 0.41 No irq handler for vector [ 4748.781941] Lustre: DEBUG MARKER: 'ls -l' done [ 4767.376155] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123aa.sanity: 17 seconds [ 4767.381123] do_IRQ: 2 callbacks suppressed [ 4767.381131] do_IRQ: 0.41 No irq handler for vector [ 4767.384292] do_IRQ: 0.41 No irq handler for vector [ 4767.388904] do_IRQ: 0.41 No irq handler for vector [ 4767.393513] do_IRQ: 0.41 No irq handler for vector [ 4767.407147] do_IRQ: 0.41 No irq handler for vector [ 4767.409018] do_IRQ: 0.41 No irq handler for vector [ 4767.415911] do_IRQ: 0.41 No irq handler for vector [ 4767.417227] do_IRQ: 0.41 No irq handler for vector [ 4767.428153] do_IRQ: 0.41 No irq handler for vector [ 4767.436725] do_IRQ: 0.41 No irq handler for vector [ 4777.683918] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 01:11:27 (1790745087) [ 4777.698944] do_IRQ: 1 callbacks suppressed [ 4777.698954] do_IRQ: 0.41 No irq handler for vector [ 4777.713149] do_IRQ: 0.41 No irq handler for vector [ 4777.725319] do_IRQ: 0.41 No irq handler for vector [ 4777.727927] do_IRQ: 0.41 No irq handler for vector [ 4777.731515] do_IRQ: 0.41 No irq handler for vector [ 4777.736735] do_IRQ: 0.41 No irq handler for vector [ 4777.739980] do_IRQ: 0.41 No irq handler for vector [ 4777.742720] do_IRQ: 0.41 No irq handler for vector [ 4777.745551] do_IRQ: 0.41 No irq handler for vector [ 4777.748395] do_IRQ: 0.41 No irq handler for vector [ 4786.698571] Lustre: DEBUG MARKER: 'statx -l' 100 files without statahead: 2 sec [ 4786.705253] do_IRQ: 1 callbacks suppressed [ 4786.705264] do_IRQ: 0.41 No irq handler for vector [ 4786.708482] do_IRQ: 0.41 No irq handler for vector [ 4786.719254] do_IRQ: 0.41 No irq handler for vector [ 4786.724336] do_IRQ: 0.41 No irq handler for vector [ 4786.726401] do_IRQ: 0.41 No irq handler for vector [ 4786.728437] do_IRQ: 0.41 No irq handler for vector [ 4786.730509] do_IRQ: 0.41 No irq handler for vector [ 4786.733405] do_IRQ: 0.41 No irq handler for vector [ 4786.736220] do_IRQ: 0.41 No irq handler for vector [ 4786.740737] do_IRQ: 0.41 No irq handler for vector [ 4789.212544] Lustre: DEBUG MARKER: 'statx -l' 100 files with statahead: 1 sec [ 4824.053390] Lustre: DEBUG MARKER: 'statx -l' 1000 files without statahead: 19 sec [ 4824.062027] do_IRQ: 3 callbacks suppressed [ 4824.062038] do_IRQ: 0.41 No irq handler for vector [ 4824.081347] do_IRQ: 0.41 No irq handler for vector [ 4824.086253] do_IRQ: 0.41 No irq handler for vector [ 4824.088858] do_IRQ: 0.41 No irq handler for vector [ 4824.093887] do_IRQ: 0.41 No irq handler for vector [ 4824.098515] do_IRQ: 0.41 No irq handler for vector [ 4824.104828] do_IRQ: 0.41 No irq handler for vector [ 4824.112832] do_IRQ: 0.41 No irq handler for vector [ 4824.121447] do_IRQ: 0.41 No irq handler for vector [ 4824.129858] do_IRQ: 0.41 No irq handler for vector [ 4831.812178] Lustre: DEBUG MARKER: 'statx -l' 1000 files with statahead: 6 sec [ 4831.817754] do_IRQ: 1 callbacks suppressed [ 4831.817763] do_IRQ: 0.41 No irq handler for vector [ 4831.826332] do_IRQ: 0.41 No irq handler for vector [ 4831.828954] do_IRQ: 0.41 No irq handler for vector [ 4831.834517] do_IRQ: 0.41 No irq handler for vector [ 4831.840426] do_IRQ: 0.41 No irq handler for vector [ 4831.845146] do_IRQ: 0.41 No irq handler for vector [ 4831.848086] do_IRQ: 0.41 No irq handler for vector [ 4831.852874] do_IRQ: 0.41 No irq handler for vector [ 4831.858908] do_IRQ: 0.41 No irq handler for vector [ 4831.862290] do_IRQ: 0.41 No irq handler for vector [ 4833.493642] Lustre: DEBUG MARKER: 'statx -l' done [ 4852.559134] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ab.sanity: 17 seconds [ 4852.579707] do_IRQ: 2 callbacks suppressed [ 4852.579716] do_IRQ: 0.41 No irq handler for vector [ 4852.592108] do_IRQ: 0.41 No irq handler for vector [ 4852.600383] do_IRQ: 0.41 No irq handler for vector [ 4852.607295] do_IRQ: 0.41 No irq handler for vector [ 4852.610232] do_IRQ: 0.41 No irq handler for vector [ 4852.614299] do_IRQ: 0.41 No irq handler for vector [ 4852.620499] do_IRQ: 0.41 No irq handler for vector [ 4852.625293] do_IRQ: 0.41 No irq handler for vector [ 4852.630445] do_IRQ: 0.41 No irq handler for vector [ 4852.634263] do_IRQ: 0.41 No irq handler for vector [ 4864.019643] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 01:12:52 (1790745172) [ 4864.034829] do_IRQ: 1 callbacks suppressed [ 4864.034838] do_IRQ: 0.41 No irq handler for vector [ 4864.048465] do_IRQ: 0.41 No irq handler for vector [ 4864.050892] do_IRQ: 0.41 No irq handler for vector [ 4864.060882] do_IRQ: 0.41 No irq handler for vector [ 4864.065011] do_IRQ: 0.41 No irq handler for vector [ 4864.071707] do_IRQ: 0.41 No irq handler for vector [ 4864.077129] do_IRQ: 0.41 No irq handler for vector [ 4864.089071] do_IRQ: 0.41 No irq handler for vector [ 4864.111293] do_IRQ: 0.41 No irq handler for vector [ 4864.112982] do_IRQ: 0.41 No irq handler for vector [ 4871.989839] Lustre: DEBUG MARKER: 'statx -c 1 [ 4871.994892] do_IRQ: 1 callbacks suppressed [ 4871.994902] do_IRQ: 0.41 No irq handler for vector [ 4872.005827] do_IRQ: 0.41 No irq handler for vector [ 4872.013342] do_IRQ: 0.41 No irq handler for vector [ 4872.029962] do_IRQ: 0.41 No irq handler for vector [ 4872.034721] do_IRQ: 0.41 No irq handler for vector [ 4872.037600] do_IRQ: 0.41 No irq handler for vector [ 4872.041148] do_IRQ: 0.41 No irq handler for vector [ 4872.044602] do_IRQ: 0.41 No irq handler for vector [ 4872.047387] do_IRQ: 0.41 No irq handler for vector [ 4872.052994] do_IRQ: 0.41 No irq handler for vector [ 4874.746978] Lustre: DEBUG MARKER: 'statx -c 1 [ 4904.263038] Lustre: DEBUG MARKER: 'statx -c 1 [ 4904.269707] do_IRQ: 2 callbacks suppressed [ 4904.269717] do_IRQ: 0.41 No irq handler for vector [ 4904.278081] do_IRQ: 0.41 No irq handler for vector [ 4904.279785] do_IRQ: 0.41 No irq handler for vector [ 4904.286622] do_IRQ: 0.41 No irq handler for vector [ 4904.296389] do_IRQ: 0.41 No irq handler for vector [ 4904.299959] do_IRQ: 0.41 No irq handler for vector [ 4904.302842] do_IRQ: 0.41 No irq handler for vector [ 4904.305508] do_IRQ: 0.41 No irq handler for vector [ 4904.307897] do_IRQ: 0.41 No irq handler for vector [ 4904.310823] do_IRQ: 0.41 No irq handler for vector [ 4909.052891] Lustre: DEBUG MARKER: 'statx -c 1 [ 4910.562131] Lustre: DEBUG MARKER: 'statx -c 1 [ 4910.565844] do_IRQ: 2 callbacks suppressed [ 4910.565855] do_IRQ: 0.41 No irq handler for vector [ 4910.574887] do_IRQ: 0.41 No irq handler for vector [ 4910.578815] do_IRQ: 0.41 No irq handler for vector [ 4910.581120] do_IRQ: 0.41 No irq handler for vector [ 4910.583751] do_IRQ: 0.41 No irq handler for vector [ 4910.586758] do_IRQ: 0.41 No irq handler for vector [ 4910.588905] do_IRQ: 0.41 No irq handler for vector [ 4910.595651] do_IRQ: 0.41 No irq handler for vector [ 4910.599770] do_IRQ: 0.41 No irq handler for vector [ 4910.611028] do_IRQ: 0.41 No irq handler for vector [ 4928.479871] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ac.sanity: 16 seconds [ 4928.484908] do_IRQ: 1 callbacks suppressed [ 4928.484918] do_IRQ: 0.41 No irq handler for vector [ 4928.497597] do_IRQ: 0.41 No irq handler for vector [ 4928.500961] do_IRQ: 0.41 No irq handler for vector [ 4928.506367] do_IRQ: 0.41 No irq handler for vector [ 4928.508960] do_IRQ: 0.41 No irq handler for vector [ 4928.511513] do_IRQ: 0.41 No irq handler for vector [ 4928.514953] do_IRQ: 0.41 No irq handler for vector [ 4928.518485] do_IRQ: 0.41 No irq handler for vector [ 4928.522348] do_IRQ: 0.41 No irq handler for vector [ 4928.524667] do_IRQ: 0.41 No irq handler for vector [ 4934.547973] Lustre: DEBUG MARKER: 'statx --cached=always -D' 100 files without statahead: 0 sec [ 4934.555303] do_IRQ: 1 callbacks suppressed [ 4934.555312] do_IRQ: 0.41 No irq handler for vector [ 4934.562780] do_IRQ: 0.41 No irq handler for vector [ 4934.567534] do_IRQ: 0.41 No irq handler for vector [ 4934.574055] do_IRQ: 0.41 No irq handler for vector [ 4934.578785] do_IRQ: 0.41 No irq handler for vector [ 4934.582210] do_IRQ: 0.41 No irq handler for vector [ 4934.587298] do_IRQ: 0.41 No irq handler for vector [ 4934.589671] do_IRQ: 0.41 No irq handler for vector [ 4934.592885] do_IRQ: 0.41 No irq handler for vector [ 4934.596240] do_IRQ: 0.41 No irq handler for vector [ 4936.397862] Lustre: DEBUG MARKER: 'statx --cached=always -D' 100 files with statahead: 0 sec [ 4953.387641] Lustre: DEBUG MARKER: 'statx --cached=always -D' 1000 files without statahead: 0 sec [ 4953.392314] do_IRQ: 2 callbacks suppressed [ 4953.392324] do_IRQ: 0.41 No irq handler for vector [ 4953.396345] do_IRQ: 0.41 No irq handler for vector [ 4953.400366] do_IRQ: 0.41 No irq handler for vector [ 4953.402143] do_IRQ: 0.41 No irq handler for vector [ 4953.410603] do_IRQ: 0.41 No irq handler for vector [ 4953.412390] do_IRQ: 0.41 No irq handler for vector [ 4953.421833] do_IRQ: 0.41 No irq handler for vector [ 4953.426762] do_IRQ: 0.41 No irq handler for vector [ 4953.431455] do_IRQ: 0.41 No irq handler for vector [ 4953.435462] do_IRQ: 0.41 No irq handler for vector [ 4955.448707] Lustre: DEBUG MARKER: 'statx --cached=always -D' 1000 files with statahead: 0 sec [ 5062.269784] Lustre: DEBUG MARKER: 'statx --cached=always -D' 10000 files without statahead: 1 sec [ 5062.281830] do_IRQ: 3 callbacks suppressed [ 5062.281840] do_IRQ: 0.41 No irq handler for vector [ 5062.285617] do_IRQ: 0.41 No irq handler for vector [ 5062.287863] do_IRQ: 0.41 No irq handler for vector [ 5062.290646] do_IRQ: 0.41 No irq handler for vector [ 5062.295330] do_IRQ: 0.41 No irq handler for vector [ 5062.305428] do_IRQ: 0.41 No irq handler for vector [ 5062.311673] do_IRQ: 0.41 No irq handler for vector [ 5062.317144] do_IRQ: 0.41 No irq handler for vector [ 5062.324776] do_IRQ: 0.41 No irq handler for vector [ 5062.328173] do_IRQ: 0.41 No irq handler for vector [ 5064.619080] Lustre: DEBUG MARKER: 'statx --cached=always -D' 10000 files with statahead: 0 sec [ 5066.164915] Lustre: DEBUG MARKER: 'statx --cached=always -D' done [ 5340.239131] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ac.sanity: 273 seconds [ 5340.242849] do_IRQ: 4 callbacks suppressed [ 5340.242860] do_IRQ: 0.41 No irq handler for vector [ 5340.261238] do_IRQ: 0.41 No irq handler for vector [ 5340.266256] do_IRQ: 0.41 No irq handler for vector [ 5340.270708] do_IRQ: 0.41 No irq handler for vector [ 5340.279155] do_IRQ: 0.41 No irq handler for vector [ 5340.283042] do_IRQ: 0.41 No irq handler for vector [ 5340.288102] do_IRQ: 0.41 No irq handler for vector [ 5340.290583] do_IRQ: 0.41 No irq handler for vector [ 5340.294269] do_IRQ: 0.41 No irq handler for vector [ 5340.296726] do_IRQ: 0.41 No irq handler for vector [ 5354.969526] Lustre: DEBUG MARKER: == sanity test 123ad: Verify batching statahead works correctly ========================================================== 01:21:03 (1790745663) [ 5354.979825] do_IRQ: 1 callbacks suppressed [ 5354.979836] do_IRQ: 0.41 No irq handler for vector [ 5355.002660] do_IRQ: 0.41 No irq handler for vector [ 5355.008560] do_IRQ: 0.41 No irq handler for vector [ 5355.015485] do_IRQ: 0.41 No irq handler for vector [ 5355.019235] do_IRQ: 0.41 No irq handler for vector [ 5355.022166] do_IRQ: 0.41 No irq handler for vector [ 5355.026230] do_IRQ: 0.41 No irq handler for vector [ 5355.027829] do_IRQ: 0.41 No irq handler for vector [ 5355.029938] do_IRQ: 0.41 No irq handler for vector [ 5355.033408] do_IRQ: 0.41 No irq handler for vector [ 5364.922903] Lustre: DEBUG MARKER: 'ls -l' 100 files without statahead: 2 sec [ 5364.926281] do_IRQ: 1 callbacks suppressed [ 5364.926290] do_IRQ: 0.41 No irq handler for vector [ 5364.941206] do_IRQ: 0.41 No irq handler for vector [ 5364.949716] do_IRQ: 0.41 No irq handler for vector [ 5364.953701] do_IRQ: 0.41 No irq handler for vector [ 5364.956787] do_IRQ: 0.41 No irq handler for vector [ 5364.960390] do_IRQ: 0.41 No irq handler for vector [ 5364.963301] do_IRQ: 0.41 No irq handler for vector [ 5364.966398] do_IRQ: 0.41 No irq handler for vector [ 5364.969817] do_IRQ: 0.41 No irq handler for vector [ 5364.975313] do_IRQ: 0.41 No irq handler for vector [ 5368.424714] Lustre: DEBUG MARKER: 'ls -l' 100 files with statahead: 1 sec [ 5405.024742] Lustre: DEBUG MARKER: 'ls -l' 1000 files without statahead: 20 sec [ 5405.028275] do_IRQ: 2 callbacks suppressed [ 5405.028284] do_IRQ: 0.41 No irq handler for vector [ 5405.030274] do_IRQ: 0.41 No irq handler for vector [ 5405.034094] do_IRQ: 0.41 No irq handler for vector [ 5405.040919] do_IRQ: 0.41 No irq handler for vector [ 5405.045673] do_IRQ: 0.41 No irq handler for vector [ 5405.057256] do_IRQ: 0.41 No irq handler for vector [ 5405.061116] do_IRQ: 0.41 No irq handler for vector [ 5405.068523] do_IRQ: 0.41 No irq handler for vector [ 5405.081288] do_IRQ: 0.41 No irq handler for vector [ 5405.089345] do_IRQ: 0.41 No irq handler for vector [ 5416.558493] Lustre: DEBUG MARKER: 'ls -l' 1000 files with statahead: 9 sec [ 5416.565037] do_IRQ: 2 callbacks suppressed [ 5416.565048] do_IRQ: 0.41 No irq handler for vector [ 5416.579668] do_IRQ: 0.41 No irq handler for vector [ 5416.584836] do_IRQ: 0.41 No irq handler for vector [ 5416.594787] do_IRQ: 0.41 No irq handler for vector [ 5416.599133] do_IRQ: 0.41 No irq handler for vector [ 5416.612218] do_IRQ: 0.41 No irq handler for vector [ 5416.617510] do_IRQ: 0.41 No irq handler for vector [ 5416.620932] do_IRQ: 0.41 No irq handler for vector [ 5416.626355] do_IRQ: 0.41 No irq handler for vector [ 5416.630266] do_IRQ: 0.41 No irq handler for vector [ 5418.094752] Lustre: DEBUG MARKER: 'ls -l' done [ 5438.305764] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds [ 5438.309573] do_IRQ: 2 callbacks suppressed [ 5438.309588] do_IRQ: 0.41 No irq handler for vector [ 5438.315919] do_IRQ: 0.41 No irq handler for vector [ 5438.320571] do_IRQ: 0.41 No irq handler for vector [ 5438.324628] do_IRQ: 0.41 No irq handler for vector [ 5438.326337] do_IRQ: 0.41 No irq handler for vector [ 5438.328121] do_IRQ: 0.41 No irq handler for vector [ 5438.330221] do_IRQ: 0.41 No irq handler for vector [ 5438.332162] do_IRQ: 0.41 No irq handler for vector [ 5438.333812] do_IRQ: 0.41 No irq handler for vector [ 5438.335147] do_IRQ: 0.41 No irq handler for vector [ 5983.461482] Lustre: DEBUG MARKER: 'ls -l' 100 files without statahead: 2 sec [ 5983.466769] do_IRQ: 1 callbacks suppressed [ 5983.466779] do_IRQ: 0.41 No irq handler for vector [ 5983.470843] do_IRQ: 0.41 No irq handler for vector [ 5983.489623] do_IRQ: 0.41 No irq handler for vector [ 5983.492363] do_IRQ: 0.41 No irq handler for vector [ 5983.498887] do_IRQ: 0.41 No irq handler for vector [ 5983.507356] do_IRQ: 0.41 No irq handler for vector [ 5983.511084] do_IRQ: 0.41 No irq handler for vector [ 5983.514046] do_IRQ: 0.41 No irq handler for vector [ 5983.516908] do_IRQ: 0.41 No irq handler for vector [ 5983.520944] do_IRQ: 0.41 No irq handler for vector [ 5986.436799] Lustre: DEBUG MARKER: 'ls -l' 100 files with statahead: 1 sec [ 6023.774941] Lustre: DEBUG MARKER: 'ls -l' 1000 files without statahead: 20 sec [ 6023.778360] do_IRQ: 2 callbacks suppressed [ 6023.778371] do_IRQ: 0.41 No irq handler for vector [ 6023.788452] do_IRQ: 0.41 No irq handler for vector [ 6023.791584] do_IRQ: 0.41 No irq handler for vector [ 6023.794123] do_IRQ: 0.41 No irq handler for vector [ 6023.796564] do_IRQ: 0.41 No irq handler for vector [ 6023.798830] do_IRQ: 0.41 No irq handler for vector [ 6023.802075] do_IRQ: 0.41 No irq handler for vector [ 6023.808138] do_IRQ: 0.41 No irq handler for vector [ 6023.812118] do_IRQ: 0.41 No irq handler for vector [ 6023.814759] do_IRQ: 0.41 No irq handler for vector [ 6032.644937] Lustre: DEBUG MARKER: 'ls -l' 1000 files with statahead: 6 sec [ 6032.658238] do_IRQ: 1 callbacks suppressed [ 6032.658246] do_IRQ: 0.41 No irq handler for vector [ 6032.666841] do_IRQ: 0.41 No irq handler for vector [ 6032.673752] do_IRQ: 0.41 No irq handler for vector [ 6032.680919] do_IRQ: 0.41 No irq handler for vector [ 6032.686379] do_IRQ: 0.41 No irq handler for vector [ 6032.689752] do_IRQ: 0.41 No irq handler for vector [ 6032.692841] do_IRQ: 0.41 No irq handler for vector [ 6032.703310] do_IRQ: 0.41 No irq handler for vector [ 6032.708682] do_IRQ: 0.41 No irq handler for vector [ 6032.711872] do_IRQ: 0.41 No irq handler for vector [ 6034.235750] Lustre: DEBUG MARKER: 'ls -l' done [ 6055.369617] Lustre: DEBUG MARKER: unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds [ 6055.379737] do_IRQ: 2 callbacks suppressed [ 6055.379743] do_IRQ: 0.41 No irq handler for vector [ 6055.384943] do_IRQ: 0.41 No irq handler for vector [ 6055.389182] do_IRQ: 0.41 No irq handler for vector [ 6055.396957] do_IRQ: 0.41 No irq handler for vector [ 6055.404300] do_IRQ: 0.41 No irq handler for vector [ 6055.407835] do_IRQ: 0.41 No irq handler for vector [ 6055.410536] do_IRQ: 0.41 No irq handler for vector [ 6055.414249] do_IRQ: 0.41 No irq handler for vector [ 6055.422723] do_IRQ: 0.41 No irq handler for vector [ 6055.427025] do_IRQ: 0.41 No irq handler for vector [ 6489.735362] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 01:39:58 (1790746798) [ 6489.745749] do_IRQ: 1 callbacks suppressed [ 6489.745760] do_IRQ: 0.41 No irq handler for vector [ 6489.759062] do_IRQ: 0.41 No irq handler for vector [ 6489.767648] do_IRQ: 0.41 No irq handler for vector [ 6489.770694] do_IRQ: 0.41 No irq handler for vector [ 6489.772803] do_IRQ: 0.41 No irq handler for vector [ 6489.775529] do_IRQ: 0.41 No irq handler for vector [ 6489.777498] do_IRQ: 0.41 No irq handler for vector [ 6489.779938] do_IRQ: 0.41 No irq handler for vector [ 6489.788547] do_IRQ: 0.41 No irq handler for vector [ 6489.791364] do_IRQ: 0.41 No irq handler for vector [ 6512.015437] Lustre: DEBUG MARKER: ls done [ 6512.022613] do_IRQ: 1 callbacks suppressed [ 6512.022624] do_IRQ: 0.41 No irq handler for vector [ 6512.031325] do_IRQ: 0.41 No irq handler for vector [ 6512.034516] do_IRQ: 0.41 No irq handler for vector [ 6512.037117] do_IRQ: 0.41 No irq handler for vector [ 6512.039690] do_IRQ: 0.41 No irq handler for vector [ 6512.043662] do_IRQ: 0.41 No irq handler for vector [ 6512.048435] do_IRQ: 0.41 No irq handler for vector [ 6512.050823] do_IRQ: 0.41 No irq handler for vector [ 6512.053080] do_IRQ: 0.41 No irq handler for vector [ 6512.055643] do_IRQ: 0.41 No irq handler for vector [ 6534.456779] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 01:40:43 (1790746843) [ 6534.471690] do_IRQ: 1 callbacks suppressed [ 6534.471699] do_IRQ: 0.41 No irq handler for vector [ 6534.474824] do_IRQ: 0.41 No irq handler for vector [ 6534.479021] do_IRQ: 0.41 No irq handler for vector [ 6534.483369] do_IRQ: 0.41 No irq handler for vector [ 6534.491195] do_IRQ: 0.41 No irq handler for vector [ 6534.497135] do_IRQ: 0.41 No irq handler for vector [ 6534.501929] do_IRQ: 0.41 No irq handler for vector [ 6534.507407] do_IRQ: 0.41 No irq handler for vector [ 6534.512146] do_IRQ: 0.41 No irq handler for vector [ 6534.515330] do_IRQ: 0.41 No irq handler for vector [ 6546.148430] Lustre: DEBUG MARKER: == sanity test 123d: Statahead on striped directories works correctly ========================================================== 01:40:54 (1790746854) [ 6546.158351] do_IRQ: 2 callbacks suppressed [ 6546.158361] do_IRQ: 0.41 No irq handler for vector [ 6546.161433] do_IRQ: 0.41 No irq handler for vector [ 6546.164827] do_IRQ: 0.41 No irq handler for vector [ 6546.167835] do_IRQ: 0.41 No irq handler for vector [ 6546.172969] do_IRQ: 0.41 No irq handler for vector [ 6546.178680] do_IRQ: 0.41 No irq handler for vector [ 6546.182890] do_IRQ: 0.41 No irq handler for vector [ 6546.186412] do_IRQ: 0.41 No irq handler for vector [ 6546.191430] do_IRQ: 0.41 No irq handler for vector [ 6546.194589] do_IRQ: 0.41 No irq handler for vector [ 6566.514427] Lustre: DEBUG MARKER: == sanity test 123e: statahead with large wide striping == 01:41:15 (1790746875) [ 6566.519083] do_IRQ: 2 callbacks suppressed [ 6566.519095] do_IRQ: 0.41 No irq handler for vector [ 6566.533201] do_IRQ: 0.41 No irq handler for vector [ 6566.540801] do_IRQ: 0.41 No irq handler for vector [ 6566.549542] do_IRQ: 0.41 No irq handler for vector [ 6566.556590] do_IRQ: 0.41 No irq handler for vector [ 6566.560898] do_IRQ: 0.41 No irq handler for vector [ 6566.563337] do_IRQ: 0.41 No irq handler for vector [ 6566.567937] do_IRQ: 0.41 No irq handler for vector [ 6566.571199] do_IRQ: 0.41 No irq handler for vector [ 6566.574186] do_IRQ: 0.41 No irq handler for vector [ 6699.733717] Lustre: 6519:0:(mdt_handler.c:4716:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 6699.746991] do_IRQ: 1 callbacks suppressed [ 6699.747006] do_IRQ: 0.41 No irq handler for vector [ 6699.749428] do_IRQ: 0.41 No irq handler for vector [ 6699.761572] do_IRQ: 0.41 No irq handler for vector [ 6699.766461] do_IRQ: 0.41 No irq handler for vector [ 6699.770349] do_IRQ: 0.41 No irq handler for vector [ 6699.773970] do_IRQ: 0.41 No irq handler for vector [ 6699.777708] do_IRQ: 0.41 No irq handler for vector [ 6699.781307] do_IRQ: 0.41 No irq handler for vector [ 6699.785161] do_IRQ: 0.41 No irq handler for vector [ 6699.788664] do_IRQ: 0.41 No irq handler for vector [ 6849.512570] Lustre: DEBUG MARKER: == sanity test 123f: Retry mechanism with large wide striping files ========================================================== 01:45:55 (1790747155) [ 6849.522398] do_IRQ: 1 callbacks suppressed [ 6849.522405] do_IRQ: 0.41 No irq handler for vector [ 6849.528870] do_IRQ: 0.41 No irq handler for vector [ 6849.530812] do_IRQ: 0.41 No irq handler for vector [ 6849.532540] do_IRQ: 0.41 No irq handler for vector [ 6849.534261] do_IRQ: 0.41 No irq handler for vector [ 6849.536419] do_IRQ: 0.41 No irq handler for vector [ 6849.538070] do_IRQ: 0.41 No irq handler for vector [ 6849.540433] do_IRQ: 0.41 No irq handler for vector [ 6849.542396] do_IRQ: 0.41 No irq handler for vector [ 6849.543742] do_IRQ: 0.41 No irq handler for vector [ 6987.791994] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6987.798529] do_IRQ: 1 callbacks suppressed [ 6987.798536] do_IRQ: 0.41 No irq handler for vector [ 6987.806888] do_IRQ: 0.41 No irq handler for vector [ 6987.812972] do_IRQ: 0.41 No irq handler for vector [ 6987.818384] do_IRQ: 0.41 No irq handler for vector [ 6987.822011] do_IRQ: 0.41 No irq handler for vector [ 6987.826794] do_IRQ: 0.41 No irq handler for vector [ 6987.829012] do_IRQ: 0.41 No irq handler for vector [ 6987.837804] do_IRQ: 0.41 No irq handler for vector [ 6987.842476] do_IRQ: 0.41 No irq handler for vector [ 6987.847899] do_IRQ: 0.41 No irq handler for vector [ 6987.988493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7246.363387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7246.371434] do_IRQ: 2 callbacks suppressed [ 7246.371439] do_IRQ: 0.41 No irq handler for vector [ 7246.377817] do_IRQ: 0.41 No irq handler for vector [ 7246.386743] do_IRQ: 0.41 No irq handler for vector [ 7246.391945] do_IRQ: 0.41 No irq handler for vector [ 7246.397517] do_IRQ: 0.41 No irq handler for vector [ 7246.402773] do_IRQ: 0.41 No irq handler for vector [ 7246.407553] do_IRQ: 0.41 No irq handler for vector [ 7246.411968] do_IRQ: 0.41 No irq handler for vector [ 7246.415259] do_IRQ: 0.41 No irq handler for vector [ 7246.420504] do_IRQ: 0.41 No irq handler for vector [ 7267.765382] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7267.772636] do_IRQ: 1 callbacks suppressed [ 7267.772646] do_IRQ: 0.41 No irq handler for vector [ 7267.774934] do_IRQ: 0.41 No irq handler for vector [ 7267.779568] do_IRQ: 0.41 No irq handler for vector [ 7267.784696] do_IRQ: 0.41 No irq handler for vector [ 7267.790839] do_IRQ: 0.41 No irq handler for vector [ 7267.796738] do_IRQ: 0.41 No irq handler for vector [ 7267.812401] do_IRQ: 0.41 No irq handler for vector [ 7267.814084] do_IRQ: 0.41 No irq handler for vector [ 7267.821696] do_IRQ: 0.41 No irq handler for vector [ 7267.829517] do_IRQ: 0.41 No irq handler for vector [ 7766.679083] Lustre: 60989:0:(osd_handler.c:2198:osd_trans_start()) lustre-MDT0000: credits 30573 > trans_max 3200 [ 7766.691449] Lustre: 60989:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 7766.692105] do_IRQ: 2 callbacks suppressed [ 7766.692110] do_IRQ: 0.41 No irq handler for vector [ 7766.704885] Lustre: 60989:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28292/0 [ 7766.704897] Lustre: 60989:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 20/166/0, punch: 0/0/0, quota 0/0/0 [ 7766.704902] Lustre: 60989:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 7766.704907] Lustre: 60989:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 7766.704912] CPU: 1 PID: 60989 Comm: mdt00_008 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 7766.704917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7766.704919] Call Trace: [ 7766.704977] ? dump_stack+0xbb/0x10e [ 7766.705010] ? osd_trans_start+0x8d3/0x9d0 [osd_ldiskfs] [ 7766.705075] ? dt_trans_start+0x1c/0x70 [obdclass] [ 7766.705225] ? top_trans_start+0x4fd/0xd00 [ptlrpc] [ 7766.707745] ? lod_declare_ref_add+0x1a/0x30 [lod] [ 7766.707814] ? dt_declare_ref_add+0x6c/0x200 [obdclass] [ 7766.707971] ? lod_trans_start+0x109/0x4c0 [lod] [ 7766.708021] ? mdd_declare_links_add+0x60/0x90 [mdd] [ 7766.708075] ? mdd_trans_start+0x18/0x30 [mdd] [ 7766.708116] ? mdd_unlink+0x7a4/0x13c0 [mdd] [ 7766.708157] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [ 7766.708278] ? mdt_reint_rec+0x139/0x2b0 [mdt] [ 7766.718673] do_IRQ: 0.41 No irq handler for vector [ 7766.729439] do_IRQ: 0.41 No irq handler for vector [ 7766.742585] ? mdt_reint_internal+0x693/0xdc0 [mdt] [ 7766.742694] ? mdt_reint+0x163/0x190 [mdt] [ 7766.742771] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 7766.743059] ? tgt_request_handle+0x575/0x1f70 [ptlrpc] [ 7766.752886] do_IRQ: 0.41 No irq handler for vector [ 7766.764071] do_IRQ: 0.41 No irq handler for vector [ 7766.776517] do_IRQ: 0.41 No irq handler for vector [ 7766.790790] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 7766.792042] do_IRQ: 0.41 No irq handler for vector [ 7766.793958] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7766.795637] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [ 7766.795914] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [ 7766.796133] ? kthread+0x1d1/0x200 [ 7766.796139] ? set_kthread_struct+0x70/0x70 [ 7766.796143] ? ret_from_fork+0x1f/0x30 [ 7767.014844] do_IRQ: 0.41 No irq handler for vector [ 7767.019214] do_IRQ: 0.41 No irq handler for vector [ 7767.020852] do_IRQ: 0.41 No irq handler for vector [ 7826.980155] Lustre: 28848:0:(osd_handler.c:2198:osd_trans_start()) lustre-MDT0000: credits 30549 > trans_max 3200 [ 7826.987231] Lustre: 28848:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 7826.987808] do_IRQ: 1 callbacks suppressed [ 7826.987813] do_IRQ: 0.41 No irq handler for vector [ 7826.993927] Lustre: 28848:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28292/0 [ 7826.993941] Lustre: 28848:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 20/142/0, punch: 0/0/0, quota 0/0/0 [ 7826.993947] Lustre: 28848:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 7826.993952] Lustre: 28848:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 7826.993958] CPU: 3 PID: 28848 Comm: mdt00_005 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 7826.993963] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7826.993966] Call Trace: [ 7826.994297] ? dump_stack+0xbb/0x10e [ 7826.996176] do_IRQ: 0.41 No irq handler for vector [ 7827.003082] do_IRQ: 0.41 No irq handler for vector [ 7827.004597] ? osd_trans_start+0x8d3/0x9d0 [osd_ldiskfs] [ 7827.004676] ? dt_trans_start+0x1c/0x70 [obdclass] [ 7827.004833] ? top_trans_start+0x4fd/0xd00 [ptlrpc] [ 7827.009546] do_IRQ: 0.41 No irq handler for vector [ 7827.022870] do_IRQ: 0.41 No irq handler for vector [ 7827.026080] ? lod_declare_ref_add+0x1a/0x30 [lod] [ 7827.026157] ? dt_declare_ref_add+0x6c/0x200 [obdclass] [ 7827.030676] do_IRQ: 0.41 No irq handler for vector [ 7827.039288] do_IRQ: 0.41 No irq handler for vector [ 7827.039630] ? lod_trans_start+0x109/0x4c0 [lod] [ 7827.039698] ? mdd_declare_links_add+0x60/0x90 [mdd] [ 7827.039749] ? mdd_trans_start+0x18/0x30 [mdd] [ 7827.039788] ? mdd_unlink+0x7a4/0x13c0 [mdd] [ 7827.039826] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [ 7827.039932] ? mdt_reint_rec+0x139/0x2b0 [mdt] [ 7827.040026] ? mdt_reint_internal+0x693/0xdc0 [mdt] [ 7827.040108] ? mdt_reint+0x163/0x190 [mdt] [ 7827.040190] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 7827.040552] ? tgt_request_handle+0x575/0x1f70 [ptlrpc] [ 7827.040783] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 7827.041029] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7827.041178] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [ 7827.041463] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [ 7827.041686] ? kthread+0x1d1/0x200 [ 7827.041692] ? set_kthread_struct+0x70/0x70 [ 7827.041695] ? ret_from_fork+0x1f/0x30 [ 7827.155586] do_IRQ: 0.41 No irq handler for vector [ 7827.160541] do_IRQ: 0.41 No irq handler for vector [ 7827.165165] do_IRQ: 0.41 No irq handler for vector [ 7956.586990] Lustre: 60991:0:(osd_handler.c:2198:osd_trans_start()) lustre-MDT0000: credits 30573 > trans_max 3200 [ 7956.592213] Lustre: 60991:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 7956.592895] do_IRQ: 1 callbacks suppressed [ 7956.592901] do_IRQ: 0.41 No irq handler for vector [ 7956.597227] Lustre: 60991:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28292/0 [ 7956.597238] Lustre: 60991:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 20/166/0, punch: 0/0/0, quota 0/0/0 [ 7956.597243] Lustre: 60991:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 7956.597249] Lustre: 60991:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 7956.597254] CPU: 1 PID: 60991 Comm: mdt00_009 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 7956.597258] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7956.597260] Call Trace: [ 7956.597314] ? dump_stack+0xbb/0x10e [ 7956.597347] ? osd_trans_start+0x8d3/0x9d0 [osd_ldiskfs] [ 7956.597415] ? dt_trans_start+0x1c/0x70 [obdclass] [ 7956.597862] ? top_trans_start+0x4fd/0xd00 [ptlrpc] [ 7956.598230] ? lod_declare_ref_add+0x1a/0x30 [lod] [ 7956.598295] ? dt_declare_ref_add+0x6c/0x200 [obdclass] [ 7956.598445] ? lod_trans_start+0x109/0x4c0 [lod] [ 7956.598499] ? mdd_declare_links_add+0x60/0x90 [mdd] [ 7956.598550] ? mdd_trans_start+0x18/0x30 [mdd] [ 7956.598696] ? mdd_unlink+0x7a4/0x13c0 [mdd] [ 7956.598741] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [ 7956.598868] ? mdt_reint_rec+0x139/0x2b0 [mdt] [ 7956.603104] do_IRQ: 0.41 No irq handler for vector [ 7956.612026] do_IRQ: 0.41 No irq handler for vector [ 7956.623393] ? mdt_reint_internal+0x693/0xdc0 [mdt] [ 7956.630064] do_IRQ: 0.41 No irq handler for vector [ 7956.635819] ? mdt_reint+0x163/0x190 [mdt] [ 7956.640981] do_IRQ: 0.41 No irq handler for vector [ 7956.654247] do_IRQ: 0.41 No irq handler for vector [ 7956.655042] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 7956.655351] ? tgt_request_handle+0x575/0x1f70 [ptlrpc] [ 7956.655572] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 7956.655800] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 7956.655957] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [ 7956.656170] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [ 7956.656388] ? kthread+0x1d1/0x200 [ 7956.656394] ? set_kthread_struct+0x70/0x70 [ 7956.656398] ? ret_from_fork+0x1f/0x30 [ 7956.758927] do_IRQ: 0.41 No irq handler for vector [ 7956.763012] do_IRQ: 0.41 No irq handler for vector [ 7956.765929] do_IRQ: 0.41 No irq handler for vector [ 7956.770370] do_IRQ: 0.41 No irq handler for vector [ 7997.128170] Lustre: DEBUG MARKER: == sanity test 123g: Test for stat-ahead advise ========== 02:05:01 (1790748301) [ 7997.144540] do_IRQ: 1 callbacks suppressed [ 7997.144547] do_IRQ: 0.41 No irq handler for vector [ 7997.153446] do_IRQ: 0.41 No irq handler for vector [ 7997.159974] do_IRQ: 0.41 No irq handler for vector [ 7997.163632] do_IRQ: 0.41 No irq handler for vector [ 7997.170204] do_IRQ: 0.41 No irq handler for vector [ 7997.176426] do_IRQ: 0.41 No irq handler for vector [ 7997.179883] do_IRQ: 0.41 No irq handler for vector [ 7997.181793] do_IRQ: 0.41 No irq handler for vector [ 7997.183441] do_IRQ: 0.41 No irq handler for vector [ 7997.186712] do_IRQ: 0.41 No irq handler for vector [ 8088.977858] Lustre: DEBUG MARKER: == sanity test 123h: Verify statahead work with the fname pattern ========================================================== 02:06:37 (1790748397) [ 8088.994351] do_IRQ: 1 callbacks suppressed [ 8088.994359] do_IRQ: 0.41 No irq handler for vector [ 8088.998528] do_IRQ: 0.41 No irq handler for vector [ 8089.004426] do_IRQ: 0.41 No irq handler for vector [ 8089.009672] do_IRQ: 0.41 No irq handler for vector [ 8089.012571] do_IRQ: 0.41 No irq handler for vector [ 8089.018197] do_IRQ: 0.41 No irq handler for vector [ 8089.020302] do_IRQ: 0.41 No irq handler for vector [ 8089.022441] do_IRQ: 0.41 No irq handler for vector [ 8089.024477] do_IRQ: 0.41 No irq handler for vector [ 8089.031824] do_IRQ: 0.41 No irq handler for vector [ 9513.709806] Lustre: DEBUG MARKER: == sanity test 123i: Verify statahead work with the fname indexing pattern ========================================================== 02:30:22 (1790749822) [ 9513.722965] do_IRQ: 1 callbacks suppressed [ 9513.722976] do_IRQ: 0.41 No irq handler for vector [ 9513.724872] do_IRQ: 0.41 No irq handler for vector [ 9513.726593] do_IRQ: 0.41 No irq handler for vector [ 9513.729840] do_IRQ: 0.41 No irq handler for vector [ 9513.733314] do_IRQ: 0.41 No irq handler for vector [ 9513.736526] do_IRQ: 0.41 No irq handler for vector [ 9513.739439] do_IRQ: 0.41 No irq handler for vector [ 9513.742485] do_IRQ: 0.41 No irq handler for vector [ 9513.745346] do_IRQ: 0.41 No irq handler for vector [ 9513.747971] do_IRQ: 0.41 No irq handler for vector [ 9631.195683] Lustre: DEBUG MARKER: == sanity test 123j: -ENOENT error from batched statahead be handled correctly ========================================================== 02:32:20 (1790749940) [ 9631.215223] do_IRQ: 2 callbacks suppressed [ 9631.215233] do_IRQ: 0.41 No irq handler for vector [ 9631.222347] do_IRQ: 0.41 No irq handler for vector [ 9631.227131] do_IRQ: 0.41 No irq handler for vector [ 9631.234288] do_IRQ: 0.41 No irq handler for vector [ 9631.240746] do_IRQ: 0.41 No irq handler for vector [ 9631.244742] do_IRQ: 0.41 No irq handler for vector [ 9631.248643] do_IRQ: 0.41 No irq handler for vector [ 9631.250375] do_IRQ: 0.41 No irq handler for vector [ 9631.254703] do_IRQ: 0.41 No irq handler for vector [ 9631.257169] do_IRQ: 0.41 No irq handler for vector [ 9638.000483] Lustre: DEBUG MARKER: == sanity test 123k: Verify statahead work with mdtest shared stat() mode ========================================================== 02:32:27 (1790749947) [ 9638.014242] do_IRQ: 2 callbacks suppressed [ 9638.014253] do_IRQ: 0.41 No irq handler for vector [ 9638.019591] do_IRQ: 0.41 No irq handler for vector [ 9638.023301] do_IRQ: 0.41 No irq handler for vector [ 9638.029824] do_IRQ: 0.41 No irq handler for vector [ 9638.032439] do_IRQ: 0.41 No irq handler for vector [ 9638.035294] do_IRQ: 0.41 No irq handler for vector [ 9638.037747] do_IRQ: 0.41 No irq handler for vector [ 9638.045585] do_IRQ: 0.41 No irq handler for vector [ 9638.049408] do_IRQ: 0.41 No irq handler for vector [ 9638.055842] do_IRQ: 0.41 No irq handler for vector [ 9639.498405] Lustre: DEBUG MARKER: SKIP: sanity test_123k mdtest not found [ 9641.306609] Lustre: DEBUG MARKER: == sanity test 123l: Avoid panic when revalidate a local cached entry ========================================================== 02:32:30 (1790749950) [ 9721.458678] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 02:33:50 (1790750030) [ 9721.473988] do_IRQ: 3 callbacks suppressed [ 9721.474006] do_IRQ: 0.41 No irq handler for vector [ 9721.479232] do_IRQ: 0.41 No irq handler for vector [ 9721.483326] do_IRQ: 0.41 No irq handler for vector [ 9721.487221] do_IRQ: 0.41 No irq handler for vector [ 9721.490442] do_IRQ: 0.41 No irq handler for vector [ 9721.493375] do_IRQ: 0.41 No irq handler for vector [ 9721.495529] do_IRQ: 0.41 No irq handler for vector [ 9721.499609] do_IRQ: 0.41 No irq handler for vector [ 9721.502685] do_IRQ: 0.41 No irq handler for vector [ 9721.503829] do_IRQ: 0.41 No irq handler for vector [ 9722.841813] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 9760.824822] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff91f71d8b9000 [ 9760.833803] do_IRQ: 3 callbacks suppressed [ 9760.833812] do_IRQ: 0.41 No irq handler for vector [ 9760.840655] do_IRQ: 0.41 No irq handler for vector [ 9760.848930] do_IRQ: 0.41 No irq handler for vector [ 9760.854427] do_IRQ: 0.41 No irq handler for vector [ 9760.859199] do_IRQ: 0.41 No irq handler for vector [ 9760.865683] do_IRQ: 0.41 No irq handler for vector [ 9760.872199] do_IRQ: 0.41 No irq handler for vector [ 9760.876890] do_IRQ: 0.41 No irq handler for vector [ 9760.883849] do_IRQ: 0.41 No irq handler for vector [ 9760.887336] do_IRQ: 0.41 No irq handler for vector [ 9762.771496] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff91f71d8b9000 [ 9764.237480] Lustre: DEBUG MARKER: LRU=1015 [ 9765.686557] Lustre: DEBUG MARKER: LIMIT=46162 [ 9767.271440] Lustre: DEBUG MARKER: LVF=5457500 [ 9767.273867] do_IRQ: 5 callbacks suppressed [ 9767.273878] do_IRQ: 0.41 No irq handler for vector [ 9767.284786] do_IRQ: 0.41 No irq handler for vector [ 9767.288492] do_IRQ: 0.41 No irq handler for vector [ 9767.290919] do_IRQ: 0.41 No irq handler for vector [ 9767.295106] do_IRQ: 0.41 No irq handler for vector [ 9767.297682] do_IRQ: 0.41 No irq handler for vector [ 9767.300530] do_IRQ: 0.41 No irq handler for vector [ 9767.302996] do_IRQ: 0.41 No irq handler for vector [ 9767.305630] do_IRQ: 0.41 No irq handler for vector [ 9767.309108] do_IRQ: 0.41 No irq handler for vector [ 9768.822682] Lustre: DEBUG MARKER: OLD_LVF=100 [ 9770.339652] Lustre: DEBUG MARKER: Sleep 50 sec [ 9822.799633] Lustre: DEBUG MARKER: Dropped 532 locks in 50s [ 9822.801735] do_IRQ: 3 callbacks suppressed [ 9822.801744] do_IRQ: 0.41 No irq handler for vector [ 9822.807972] do_IRQ: 0.41 No irq handler for vector [ 9822.810966] do_IRQ: 0.41 No irq handler for vector [ 9822.813512] do_IRQ: 0.41 No irq handler for vector [ 9822.817885] do_IRQ: 0.41 No irq handler for vector [ 9822.820228] do_IRQ: 0.41 No irq handler for vector [ 9822.824584] do_IRQ: 0.41 No irq handler for vector [ 9822.828106] do_IRQ: 0.41 No irq handler for vector [ 9822.832808] do_IRQ: 0.41 No irq handler for vector [ 9822.835929] do_IRQ: 0.41 No irq handler for vector [ 9824.307508] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 9849.128912] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 02:35:58 (1790750158) [ 9849.151750] do_IRQ: 2 callbacks suppressed [ 9849.151760] do_IRQ: 0.41 No irq handler for vector [ 9849.159356] do_IRQ: 0.41 No irq handler for vector [ 9849.170256] do_IRQ: 0.41 No irq handler for vector [ 9849.181969] do_IRQ: 0.41 No irq handler for vector [ 9849.187606] do_IRQ: 0.41 No irq handler for vector [ 9849.192824] do_IRQ: 0.41 No irq handler for vector [ 9849.195599] do_IRQ: 0.41 No irq handler for vector [ 9849.198363] do_IRQ: 0.41 No irq handler for vector [ 9849.201176] do_IRQ: 0.41 No irq handler for vector [ 9849.203755] do_IRQ: 0.41 No irq handler for vector [ 9946.638294] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 9946.643016] do_IRQ: 1 callbacks suppressed [ 9946.643027] do_IRQ: 0.41 No irq handler for vector [ 9946.649016] do_IRQ: 0.41 No irq handler for vector [ 9946.656606] do_IRQ: 0.41 No irq handler for vector [ 9946.662791] do_IRQ: 0.41 No irq handler for vector [ 9946.665176] do_IRQ: 0.41 No irq handler for vector [ 9946.667957] do_IRQ: 0.41 No irq handler for vector [ 9946.669915] do_IRQ: 0.41 No irq handler for vector [ 9946.674417] do_IRQ: 0.41 No irq handler for vector [ 9946.677876] do_IRQ: 0.41 No irq handler for vector [ 9946.682294] do_IRQ: 0.41 No irq handler for vector [10113.447702] Lustre: DEBUG MARKER: ls -la time: 165 seconds [10113.453902] do_IRQ: 2 callbacks suppressed [10113.453914] do_IRQ: 0.41 No irq handler for vector [10113.465122] do_IRQ: 0.41 No irq handler for vector [10113.471890] do_IRQ: 0.41 No irq handler for vector [10113.474462] do_IRQ: 0.41 No irq handler for vector [10113.476713] do_IRQ: 0.41 No irq handler for vector [10113.479064] do_IRQ: 0.41 No irq handler for vector [10113.481462] do_IRQ: 0.41 No irq handler for vector [10113.486330] do_IRQ: 0.41 No irq handler for vector [10113.488477] do_IRQ: 0.41 No irq handler for vector [10113.492333] do_IRQ: 0.41 No irq handler for vector [10114.974330] Lustre: DEBUG MARKER: lru_size = 400 [10295.818571] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [10295.823269] do_IRQ: 2 callbacks suppressed [10295.823279] do_IRQ: 0.41 No irq handler for vector [10295.835443] do_IRQ: 0.41 No irq handler for vector [10295.845755] do_IRQ: 0.41 No irq handler for vector [10295.849137] do_IRQ: 0.41 No irq handler for vector [10295.853944] do_IRQ: 0.41 No irq handler for vector [10295.857149] do_IRQ: 0.41 No irq handler for vector [10295.863840] do_IRQ: 0.41 No irq handler for vector [10295.870823] do_IRQ: 0.41 No irq handler for vector [10295.874861] do_IRQ: 0.41 No irq handler for vector [10295.880585] do_IRQ: 0.41 No irq handler for vector [10369.469280] Lustre: DEBUG MARKER: ls -la time: 69 seconds [10369.480094] do_IRQ: 1 callbacks suppressed [10369.480101] do_IRQ: 0.41 No irq handler for vector [10369.483776] do_IRQ: 0.41 No irq handler for vector [10369.487092] do_IRQ: 0.41 No irq handler for vector [10369.493901] do_IRQ: 0.41 No irq handler for vector [10369.497553] do_IRQ: 0.41 No irq handler for vector [10369.501115] do_IRQ: 0.41 No irq handler for vector [10369.503561] do_IRQ: 0.41 No irq handler for vector [10369.506967] do_IRQ: 0.41 No irq handler for vector [10369.511344] do_IRQ: 0.41 No irq handler for vector [10369.515715] do_IRQ: 0.41 No irq handler for vector [10370.920694] Lustre: DEBUG MARKER: lru_size = 4006 [10372.100958] Lustre: DEBUG MARKER: ls -la is 58% faster with lru resize enabled [10435.863501] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 02:45:43 (1790750743) [10435.868931] do_IRQ: 3 callbacks suppressed [10435.868939] do_IRQ: 0.41 No irq handler for vector [10435.882572] do_IRQ: 0.41 No irq handler for vector [10435.888797] do_IRQ: 0.41 No irq handler for vector [10435.894638] do_IRQ: 0.41 No irq handler for vector [10435.901478] do_IRQ: 0.41 No irq handler for vector [10435.904541] do_IRQ: 0.41 No irq handler for vector [10435.912601] do_IRQ: 0.41 No irq handler for vector [10435.917853] do_IRQ: 0.41 No irq handler for vector [10435.928618] do_IRQ: 0.41 No irq handler for vector [10435.930314] do_IRQ: 0.41 No irq handler for vector [10467.395164] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize disabled ========================================================== 02:46:16 (1790750776) [10467.404091] do_IRQ: 1 callbacks suppressed [10467.404099] do_IRQ: 0.41 No irq handler for vector [10467.409227] do_IRQ: 0.41 No irq handler for vector [10467.411388] do_IRQ: 0.41 No irq handler for vector [10467.413839] do_IRQ: 0.41 No irq handler for vector [10467.416239] do_IRQ: 0.41 No irq handler for vector [10467.419572] do_IRQ: 0.41 No irq handler for vector [10467.424827] do_IRQ: 0.41 No irq handler for vector [10467.435568] do_IRQ: 0.41 No irq handler for vector [10467.439828] do_IRQ: 0.41 No irq handler for vector [10467.441658] do_IRQ: 0.41 No irq handler for vector [10499.607255] Lustre: DEBUG MARKER: == sanity test 124e: LFRU keep priv locks from eviction == 02:46:48 (1790750808) [10499.618695] do_IRQ: 1 callbacks suppressed [10499.618707] do_IRQ: 0.41 No irq handler for vector [10499.635147] do_IRQ: 0.41 No irq handler for vector [10499.637715] do_IRQ: 0.41 No irq handler for vector [10499.639507] do_IRQ: 0.41 No irq handler for vector [10499.648066] do_IRQ: 0.41 No irq handler for vector [10499.655467] do_IRQ: 0.41 No irq handler for vector [10499.657778] do_IRQ: 0.41 No irq handler for vector [10499.662305] do_IRQ: 0.41 No irq handler for vector [10499.669172] do_IRQ: 0.41 No irq handler for vector [10499.670943] do_IRQ: 0.41 No irq handler for vector [10564.080214] Lustre: DEBUG MARKER: == sanity test 124f: LFRU priv threshold inc/dec adjustment ========================================================== 02:47:53 (1790750873) [10564.097986] do_IRQ: 1 callbacks suppressed [10564.098009] do_IRQ: 0.41 No irq handler for vector [10564.114867] do_IRQ: 0.41 No irq handler for vector [10564.120058] do_IRQ: 0.41 No irq handler for vector [10564.123804] do_IRQ: 0.41 No irq handler for vector [10564.129811] do_IRQ: 0.41 No irq handler for vector [10564.132491] do_IRQ: 0.41 No irq handler for vector [10564.134191] do_IRQ: 0.41 No irq handler for vector [10564.139214] do_IRQ: 0.41 No irq handler for vector [10564.145045] do_IRQ: 0.41 No irq handler for vector [10564.153077] do_IRQ: 0.41 No irq handler for vector [10678.112392] Lustre: DEBUG MARKER: == sanity test 124g: LFRU performance test =============== 02:49:47 (1790750987) [10678.118151] do_IRQ: 1 callbacks suppressed [10678.118163] do_IRQ: 0.41 No irq handler for vector [10678.128667] do_IRQ: 0.41 No irq handler for vector [10678.130352] do_IRQ: 0.41 No irq handler for vector [10678.134971] do_IRQ: 0.41 No irq handler for vector [10678.138427] do_IRQ: 0.41 No irq handler for vector [10678.141814] do_IRQ: 0.41 No irq handler for vector [10678.145294] do_IRQ: 0.41 No irq handler for vector [10678.147414] do_IRQ: 0.41 No irq handler for vector [10678.150532] do_IRQ: 0.41 No irq handler for vector [10678.153547] do_IRQ: 0.41 No irq handler for vector [11542.196720] Lustre: DEBUG MARKER: == sanity test 124h: LFRU sample window size ============= 03:04:11 (1790751851) [11542.206749] do_IRQ: 1 callbacks suppressed [11542.206756] do_IRQ: 0.41 No irq handler for vector [11542.217675] do_IRQ: 0.41 No irq handler for vector [11542.229508] do_IRQ: 0.41 No irq handler for vector [11542.231398] do_IRQ: 0.41 No irq handler for vector [11542.236549] do_IRQ: 0.41 No irq handler for vector [11542.250303] do_IRQ: 0.41 No irq handler for vector [11542.258311] do_IRQ: 0.41 No irq handler for vector [11542.263242] do_IRQ: 0.41 No irq handler for vector [11542.264843] do_IRQ: 0.41 No irq handler for vector [11542.274475] do_IRQ: 0.41 No irq handler for vector [11633.387299] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 03:05:42 (1790751942) [11633.402504] do_IRQ: 1 callbacks suppressed [11633.402515] do_IRQ: 0.41 No irq handler for vector [11633.413162] do_IRQ: 0.41 No irq handler for vector [11633.419442] do_IRQ: 0.41 No irq handler for vector [11633.423990] do_IRQ: 0.41 No irq handler for vector [11633.431411] do_IRQ: 0.41 No irq handler for vector [11633.435599] do_IRQ: 0.41 No irq handler for vector [11633.441357] do_IRQ: 0.41 No irq handler for vector [11633.444106] do_IRQ: 0.41 No irq handler for vector [11633.450745] do_IRQ: 0.41 No irq handler for vector [11633.458288] do_IRQ: 0.41 No irq handler for vector [11642.750084] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 03:05:51 (1790751951) [11642.766586] do_IRQ: 1 callbacks suppressed [11642.766595] do_IRQ: 0.41 No irq handler for vector [11642.769432] do_IRQ: 0.41 No irq handler for vector [11642.772777] do_IRQ: 0.41 No irq handler for vector [11642.775826] do_IRQ: 0.41 No irq handler for vector [11642.777835] do_IRQ: 0.41 No irq handler for vector [11642.780573] do_IRQ: 0.41 No irq handler for vector [11642.783511] do_IRQ: 0.41 No irq handler for vector [11642.787603] do_IRQ: 0.41 No irq handler for vector [11642.790415] do_IRQ: 0.41 No irq handler for vector [11642.793537] do_IRQ: 0.41 No irq handler for vector [11649.241302] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 03:05:58 (1790751958) [11649.247103] do_IRQ: 2 callbacks suppressed [11649.247113] do_IRQ: 0.41 No irq handler for vector [11649.249144] do_IRQ: 0.41 No irq handler for vector [11649.254197] do_IRQ: 0.41 No irq handler for vector [11649.257347] do_IRQ: 0.41 No irq handler for vector [11649.260044] do_IRQ: 0.41 No irq handler for vector [11649.262535] do_IRQ: 0.41 No irq handler for vector [11649.265295] do_IRQ: 0.41 No irq handler for vector [11649.267400] do_IRQ: 0.41 No irq handler for vector [11649.269919] do_IRQ: 0.41 No irq handler for vector [11649.273299] do_IRQ: 0.41 No irq handler for vector [11657.233644] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 03:06:05 (1790751965) [11657.241537] do_IRQ: 2 callbacks suppressed [11657.241547] do_IRQ: 0.41 No irq handler for vector [11657.243786] do_IRQ: 0.41 No irq handler for vector [11657.249390] do_IRQ: 0.41 No irq handler for vector [11657.252978] do_IRQ: 0.41 No irq handler for vector [11657.255443] do_IRQ: 0.41 No irq handler for vector [11657.258420] do_IRQ: 0.41 No irq handler for vector [11657.260900] do_IRQ: 0.41 No irq handler for vector [11657.263766] do_IRQ: 0.41 No irq handler for vector [11657.266366] do_IRQ: 0.41 No irq handler for vector [11657.269187] do_IRQ: 0.41 No irq handler for vector [11665.436380] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [11665.446299] do_IRQ: 2 callbacks suppressed [11665.446307] do_IRQ: 0.41 No irq handler for vector [11665.452238] do_IRQ: 0.41 No irq handler for vector [11665.458902] do_IRQ: 0.41 No irq handler for vector [11665.461551] do_IRQ: 0.41 No irq handler for vector [11665.464729] do_IRQ: 0.41 No irq handler for vector [11665.467115] do_IRQ: 0.41 No irq handler for vector [11665.472204] do_IRQ: 0.41 No irq handler for vector [11665.481430] do_IRQ: 0.41 No irq handler for vector [11665.487379] do_IRQ: 0.41 No irq handler for vector [11665.496025] do_IRQ: 0.41 No irq handler for vector [11695.715311] Lustre: DEBUG MARKER: == sanity test 127d: OSC RPC latency histograms for read and write latency ========================================================== 03:06:44 (1790752004) [11695.732932] do_IRQ: 1 callbacks suppressed [11695.732941] do_IRQ: 0.41 No irq handler for vector [11695.736485] do_IRQ: 0.41 No irq handler for vector [11695.744389] do_IRQ: 0.41 No irq handler for vector [11695.749557] do_IRQ: 0.41 No irq handler for vector [11695.760135] do_IRQ: 0.41 No irq handler for vector [11695.767776] do_IRQ: 0.41 No irq handler for vector [11695.771324] do_IRQ: 0.41 No irq handler for vector [11695.774752] do_IRQ: 0.41 No irq handler for vector [11695.777710] do_IRQ: 0.41 No irq handler for vector [11695.782855] do_IRQ: 0.41 No irq handler for vector [11706.949208] Lustre: DEBUG MARKER: == sanity test 127e: client IO latency histograms by size ========================================================== 03:06:56 (1790752016) [11706.963711] do_IRQ: 2 callbacks suppressed [11706.963722] do_IRQ: 0.41 No irq handler for vector [11706.966347] do_IRQ: 0.41 No irq handler for vector [11706.974889] do_IRQ: 0.41 No irq handler for vector [11706.979251] do_IRQ: 0.41 No irq handler for vector [11706.981773] do_IRQ: 0.41 No irq handler for vector [11706.984614] do_IRQ: 0.41 No irq handler for vector [11706.992984] do_IRQ: 0.41 No irq handler for vector [11706.995849] do_IRQ: 0.41 No irq handler for vector [11706.999462] do_IRQ: 0.41 No irq handler for vector [11707.002556] do_IRQ: 0.41 No irq handler for vector [11724.304690] Lustre: DEBUG MARKER: == sanity test 127f: OST IO latency histograms by size === 03:07:13 (1790752033) [11724.309685] do_IRQ: 1 callbacks suppressed [11724.309696] do_IRQ: 0.41 No irq handler for vector [11724.319437] do_IRQ: 0.41 No irq handler for vector [11724.321270] do_IRQ: 0.41 No irq handler for vector [11724.324984] do_IRQ: 0.41 No irq handler for vector [11724.333028] do_IRQ: 0.41 No irq handler for vector [11724.336925] do_IRQ: 0.41 No irq handler for vector [11724.340084] do_IRQ: 0.41 No irq handler for vector [11724.343383] do_IRQ: 0.41 No irq handler for vector [11724.348552] do_IRQ: 0.41 No irq handler for vector [11724.351330] do_IRQ: 0.41 No irq handler for vector [11746.582266] Lustre: DEBUG MARKER: == sanity test 127g: cached_read_bytes tracks page cache hits ========================================================== 03:07:35 (1790752055) [11746.597854] do_IRQ: 1 callbacks suppressed [11746.597864] do_IRQ: 0.41 No irq handler for vector [11746.599161] do_IRQ: 0.41 No irq handler for vector [11746.604119] do_IRQ: 0.41 No irq handler for vector [11746.606284] do_IRQ: 0.41 No irq handler for vector [11746.613791] do_IRQ: 0.41 No irq handler for vector [11746.625133] do_IRQ: 0.41 No irq handler for vector [11746.627744] do_IRQ: 0.41 No irq handler for vector [11746.633498] do_IRQ: 0.41 No irq handler for vector [11746.635666] do_IRQ: 0.41 No irq handler for vector [11746.639283] do_IRQ: 0.41 No irq handler for vector [11759.059327] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 03:07:47 (1790752067) [11759.067680] do_IRQ: 2 callbacks suppressed [11759.067691] do_IRQ: 0.41 No irq handler for vector [11759.071386] do_IRQ: 0.41 No irq handler for vector [11759.076513] do_IRQ: 0.41 No irq handler for vector [11759.082906] do_IRQ: 0.41 No irq handler for vector [11759.085465] do_IRQ: 0.41 No irq handler for vector [11759.094963] do_IRQ: 0.41 No irq handler for vector [11759.100967] do_IRQ: 0.41 No irq handler for vector [11759.103803] do_IRQ: 0.41 No irq handler for vector [11759.111093] do_IRQ: 0.41 No irq handler for vector [11759.113511] do_IRQ: 0.41 No irq handler for vector [11765.469505] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 03:07:54 (1790752074) [11765.480957] do_IRQ: 2 callbacks suppressed [11765.480964] do_IRQ: 0.41 No irq handler for vector [11765.488652] do_IRQ: 0.41 No irq handler for vector [11765.498415] do_IRQ: 0.41 No irq handler for vector [11765.500225] do_IRQ: 0.41 No irq handler for vector [11765.502815] do_IRQ: 0.41 No irq handler for vector [11765.505024] do_IRQ: 0.41 No irq handler for vector [11765.506736] do_IRQ: 0.41 No irq handler for vector [11765.508455] do_IRQ: 0.41 No irq handler for vector [11765.510375] do_IRQ: 0.41 No irq handler for vector [11765.512788] do_IRQ: 0.41 No irq handler for vector [11786.549387] Lustre: 60991:0:(osd_handler.c:615:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513267, FID: [0x20000040c:0x94cf:0x0]) is approaching max size limit [11786.576297] do_IRQ: 1 callbacks suppressed [11786.576306] do_IRQ: 0.41 No irq handler for vector [11786.593755] do_IRQ: 0.41 No irq handler for vector [11786.595868] do_IRQ: 0.41 No irq handler for vector [11786.597832] do_IRQ: 0.41 No irq handler for vector [11786.599924] do_IRQ: 0.41 No irq handler for vector [11786.602585] do_IRQ: 0.41 No irq handler for vector [11786.606294] do_IRQ: 0.41 No irq handler for vector [11786.610142] do_IRQ: 0.41 No irq handler for vector [11786.621401] do_IRQ: 0.41 No irq handler for vector [11786.627604] do_IRQ: 0.41 No irq handler for vector [11788.089951] Lustre: 6514:0:(osd_handler.c:615:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513267, FID: [0x20000040c:0x94cf:0x0]) is approaching max size limit [11800.894887] Lustre: 6515:0:(osd_handler.c:611:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513267, FID: [0x20000040c:0x94cf:0x0]) has reached max size limit [11800.909285] do_IRQ: 2 callbacks suppressed [11800.909292] do_IRQ: 0.41 No irq handler for vector [11800.913581] do_IRQ: 0.41 No irq handler for vector [11800.923661] do_IRQ: 0.41 No irq handler for vector [11800.934139] do_IRQ: 0.41 No irq handler for vector [11800.948819] do_IRQ: 0.41 No irq handler for vector [11800.953595] do_IRQ: 0.41 No irq handler for vector [11800.959476] do_IRQ: 0.41 No irq handler for vector [11800.964363] do_IRQ: 0.41 No irq handler for vector [11800.970853] do_IRQ: 0.41 No irq handler for vector [11800.973757] do_IRQ: 0.41 No irq handler for vector [11834.332435] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 03:09:03 (1790752143) [11834.342834] do_IRQ: 1 callbacks suppressed [11834.342842] do_IRQ: 0.41 No irq handler for vector [11834.350432] do_IRQ: 0.41 No irq handler for vector [11834.358444] do_IRQ: 0.41 No irq handler for vector [11834.364543] do_IRQ: 0.41 No irq handler for vector [11834.370670] do_IRQ: 0.41 No irq handler for vector [11834.374593] do_IRQ: 0.41 No irq handler for vector [11834.381776] do_IRQ: 0.41 No irq handler for vector [11834.387566] do_IRQ: 0.41 No irq handler for vector [11834.396173] do_IRQ: 0.41 No irq handler for vector [11834.405309] do_IRQ: 0.41 No irq handler for vector [11841.585812] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 03:09:10 (1790752150) [11841.592787] do_IRQ: 1 callbacks suppressed [11841.592797] do_IRQ: 0.41 No irq handler for vector [11841.596877] do_IRQ: 0.41 No irq handler for vector [11841.600413] do_IRQ: 0.41 No irq handler for vector [11841.602938] do_IRQ: 0.41 No irq handler for vector [11841.608294] do_IRQ: 0.41 No irq handler for vector [11841.611253] do_IRQ: 0.41 No irq handler for vector [11841.613750] do_IRQ: 0.41 No irq handler for vector [11841.616367] do_IRQ: 0.41 No irq handler for vector [11841.619672] do_IRQ: 0.41 No irq handler for vector [11841.622431] do_IRQ: 0.41 No irq handler for vector [11850.512917] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 03:09:19 (1790752159) [11850.531703] do_IRQ: 2 callbacks suppressed [11850.531713] do_IRQ: 0.41 No irq handler for vector [11850.549772] do_IRQ: 0.41 No irq handler for vector [11850.554993] do_IRQ: 0.41 No irq handler for vector [11850.560747] do_IRQ: 0.41 No irq handler for vector [11850.566902] do_IRQ: 0.41 No irq handler for vector [11850.571048] do_IRQ: 0.41 No irq handler for vector [11850.577430] do_IRQ: 0.41 No irq handler for vector [11850.583910] do_IRQ: 0.41 No irq handler for vector [11850.589500] do_IRQ: 0.41 No irq handler for vector [11850.594624] do_IRQ: 0.41 No irq handler for vector [11858.956412] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 03:09:27 (1790752167) [11858.968409] do_IRQ: 1 callbacks suppressed [11858.968417] do_IRQ: 0.41 No irq handler for vector [11858.986390] do_IRQ: 0.41 No irq handler for vector [11858.989451] do_IRQ: 0.41 No irq handler for vector [11858.995438] do_IRQ: 0.41 No irq handler for vector [11858.998226] do_IRQ: 0.41 No irq handler for vector [11859.001204] do_IRQ: 0.41 No irq handler for vector [11859.004518] do_IRQ: 0.41 No irq handler for vector [11859.007932] do_IRQ: 0.41 No irq handler for vector [11859.011316] do_IRQ: 0.41 No irq handler for vector [11859.015356] do_IRQ: 0.41 No irq handler for vector [11860.918916] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [11862.678844] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 03:09:31 (1790752171) [11922.877577] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 03:10:31 (1790752231) [11922.886907] do_IRQ: 3 callbacks suppressed [11922.886919] do_IRQ: 0.41 No irq handler for vector [11922.895764] do_IRQ: 0.41 No irq handler for vector [11922.915739] do_IRQ: 0.41 No irq handler for vector [11922.918636] do_IRQ: 0.41 No irq handler for vector [11922.922937] do_IRQ: 0.41 No irq handler for vector [11922.926996] do_IRQ: 0.41 No irq handler for vector [11922.933617] do_IRQ: 0.41 No irq handler for vector [11922.936054] do_IRQ: 0.41 No irq handler for vector [11922.937866] do_IRQ: 0.41 No irq handler for vector [11922.944235] do_IRQ: 0.41 No irq handler for vector [11929.853786] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 03:10:38 (1790752238) [11929.871756] do_IRQ: 1 callbacks suppressed [11929.871769] do_IRQ: 0.41 No irq handler for vector [11929.879665] do_IRQ: 0.41 No irq handler for vector [11929.884414] do_IRQ: 0.41 No irq handler for vector [11929.888662] do_IRQ: 0.41 No irq handler for vector [11929.892614] do_IRQ: 0.41 No irq handler for vector [11929.898533] do_IRQ: 0.41 No irq handler for vector [11929.903611] do_IRQ: 0.41 No irq handler for vector [11929.908056] do_IRQ: 0.41 No irq handler for vector [11929.913625] do_IRQ: 0.41 No irq handler for vector [11929.919996] do_IRQ: 0.41 No irq handler for vector [11967.604716] Lustre: 28848:0:(osd_handler.c:2198:osd_trans_start()) lustre-MDT0000: credits 6549 > trans_max 3200 [11967.610889] do_IRQ: 2 callbacks suppressed [11967.610900] do_IRQ: 0.41 No irq handler for vector [11967.613310] Lustre: 28848:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [11967.613321] Lustre: 28848:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 403/403/0, xattr_set: 604/5892/0 [11967.613328] Lustre: 28848:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 20/142/0, punch: 0/0/0, quota 0/0/0 [11967.613332] Lustre: 28848:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [11967.613337] Lustre: 28848:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [11967.613341] CPU: 1 PID: 28848 Comm: mdt00_005 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [11967.613345] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11967.613347] Call Trace: [11967.613400] ? dump_stack+0xbb/0x10e [11967.613422] ? osd_trans_start+0x8d3/0x9d0 [osd_ldiskfs] [11967.613694] ? dt_trans_start+0x1c/0x70 [obdclass] [11967.613858] ? top_trans_start+0x4fd/0xd00 [ptlrpc] [11967.623376] do_IRQ: 0.41 No irq handler for vector [11967.629717] do_IRQ: 0.41 No irq handler for vector [11967.641247] ? lod_declare_ref_add+0x1a/0x30 [lod] [11967.641391] ? dt_declare_ref_add+0x6c/0x200 [obdclass] [11967.641547] ? lod_trans_start+0x109/0x4c0 [lod] [11967.641598] ? mdd_declare_links_add+0x60/0x90 [mdd] [11967.641647] ? mdd_trans_start+0x18/0x30 [mdd] [11967.652556] do_IRQ: 0.41 No irq handler for vector [11967.668628] do_IRQ: 0.41 No irq handler for vector [11967.669734] ? mdd_unlink+0x7a4/0x13c0 [mdd] [11967.669792] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [11967.669902] ? mdt_reint_rec+0x139/0x2b0 [mdt] [11967.669982] ? mdt_reint_internal+0x693/0xdc0 [mdt] [11967.676799] do_IRQ: 0.41 No irq handler for vector [11967.682605] ? mdt_reint+0x163/0x190 [mdt] [11967.688543] do_IRQ: 0.41 No irq handler for vector [11967.689451] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [11967.696443] do_IRQ: 0.41 No irq handler for vector [11967.698423] ? tgt_request_handle+0x575/0x1f70 [ptlrpc] [11967.698689] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [11967.698904] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [11967.699051] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [11967.699257] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [11967.704670] do_IRQ: 0.41 No irq handler for vector [11967.708008] do_IRQ: 0.41 No irq handler for vector [11967.709877] ? kthread+0x1d1/0x200 [11967.709891] ? set_kthread_struct+0x70/0x70 [11967.709894] ? ret_from_fork+0x1f/0x30 [11975.041159] Lustre: DEBUG MARKER: == sanity test 130h: FIEMAP deadlock ===================== 03:11:23 (1790752283) [11975.047789] do_IRQ: 1 callbacks suppressed [11975.047796] do_IRQ: 0.41 No irq handler for vector [11975.056910] do_IRQ: 0.41 No irq handler for vector [11975.060532] do_IRQ: 0.41 No irq handler for vector [11975.069392] do_IRQ: 0.41 No irq handler for vector [11975.072333] do_IRQ: 0.41 No irq handler for vector [11975.077176] do_IRQ: 0.41 No irq handler for vector [11975.080945] do_IRQ: 0.41 No irq handler for vector [11975.083937] do_IRQ: 0.41 No irq handler for vector [11975.092166] do_IRQ: 0.41 No irq handler for vector [11975.096753] do_IRQ: 0.41 No irq handler for vector [11989.125311] Lustre: DEBUG MARKER: == sanity test 130i: FIEMAP (DoM file) =================== 03:11:38 (1790752298) [11989.130298] do_IRQ: 1 callbacks suppressed [11989.130309] do_IRQ: 0.41 No irq handler for vector [11989.141329] do_IRQ: 0.41 No irq handler for vector [11989.148492] do_IRQ: 0.41 No irq handler for vector [11989.151644] do_IRQ: 0.41 No irq handler for vector [11989.154611] do_IRQ: 0.41 No irq handler for vector [11989.157048] do_IRQ: 0.41 No irq handler for vector [11989.159244] do_IRQ: 0.41 No irq handler for vector [11989.166152] do_IRQ: 0.41 No irq handler for vector [11989.170257] do_IRQ: 0.41 No irq handler for vector [11989.173436] do_IRQ: 0.41 No irq handler for vector [12032.751193] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 03:12:21 (1790752341) [12032.760635] do_IRQ: 1 callbacks suppressed [12032.760642] do_IRQ: 0.41 No irq handler for vector [12032.767820] do_IRQ: 0.41 No irq handler for vector [12032.769632] do_IRQ: 0.41 No irq handler for vector [12032.774722] do_IRQ: 0.41 No irq handler for vector [12032.784285] do_IRQ: 0.41 No irq handler for vector [12032.794616] do_IRQ: 0.41 No irq handler for vector [12032.798393] do_IRQ: 0.41 No irq handler for vector [12032.803498] do_IRQ: 0.41 No irq handler for vector [12032.805075] do_IRQ: 0.41 No irq handler for vector [12032.813082] do_IRQ: 0.41 No irq handler for vector [12040.829151] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 03:12:29 (1790752349) [12040.839679] do_IRQ: 1 callbacks suppressed [12040.839688] do_IRQ: 0.41 No irq handler for vector [12040.843232] do_IRQ: 0.41 No irq handler for vector [12040.849359] do_IRQ: 0.41 No irq handler for vector [12040.853214] do_IRQ: 0.41 No irq handler for vector [12040.857221] do_IRQ: 0.41 No irq handler for vector [12040.860261] do_IRQ: 0.41 No irq handler for vector [12040.862825] do_IRQ: 0.41 No irq handler for vector [12040.865748] do_IRQ: 0.41 No irq handler for vector [12040.873374] do_IRQ: 0.41 No irq handler for vector [12040.876637] do_IRQ: 0.41 No irq handler for vector [12048.527188] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 03:12:37 (1790752357) [12048.535236] do_IRQ: 2 callbacks suppressed [12048.535243] do_IRQ: 0.41 No irq handler for vector [12048.540384] do_IRQ: 0.41 No irq handler for vector [12048.543872] do_IRQ: 0.41 No irq handler for vector [12048.546625] do_IRQ: 0.41 No irq handler for vector [12048.549530] do_IRQ: 0.41 No irq handler for vector [12048.552533] do_IRQ: 0.41 No irq handler for vector [12048.555981] do_IRQ: 0.41 No irq handler for vector [12048.558623] do_IRQ: 0.41 No irq handler for vector [12048.561196] do_IRQ: 0.41 No irq handler for vector [12048.564625] do_IRQ: 0.41 No irq handler for vector [12055.689472] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 03:12:44 (1790752364) [12055.698721] do_IRQ: 1 callbacks suppressed [12055.698731] do_IRQ: 0.41 No irq handler for vector [12055.701705] do_IRQ: 0.41 No irq handler for vector [12055.707452] do_IRQ: 0.41 No irq handler for vector [12055.713776] do_IRQ: 0.41 No irq handler for vector [12055.716393] do_IRQ: 0.41 No irq handler for vector [12055.718311] do_IRQ: 0.41 No irq handler for vector [12055.720476] do_IRQ: 0.41 No irq handler for vector [12055.723381] do_IRQ: 0.41 No irq handler for vector [12055.726445] do_IRQ: 0.41 No irq handler for vector [12055.729146] do_IRQ: 0.41 No irq handler for vector [12062.213076] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 03:12:51 (1790752371) [12062.217400] do_IRQ: 2 callbacks suppressed [12062.217410] do_IRQ: 0.41 No irq handler for vector [12062.218658] do_IRQ: 0.41 No irq handler for vector [12062.224786] do_IRQ: 0.41 No irq handler for vector [12062.228510] do_IRQ: 0.41 No irq handler for vector [12062.231421] do_IRQ: 0.41 No irq handler for vector [12062.234350] do_IRQ: 0.41 No irq handler for vector [12062.238402] do_IRQ: 0.41 No irq handler for vector [12062.248393] do_IRQ: 0.41 No irq handler for vector [12062.255199] do_IRQ: 0.41 No irq handler for vector [12062.258406] do_IRQ: 0.41 No irq handler for vector [12069.085390] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 03:12:58 (1790752378) [12069.096882] do_IRQ: 2 callbacks suppressed [12069.096893] do_IRQ: 0.41 No irq handler for vector [12069.106286] do_IRQ: 0.41 No irq handler for vector [12069.108730] do_IRQ: 0.41 No irq handler for vector [12069.111874] do_IRQ: 0.41 No irq handler for vector [12069.115425] do_IRQ: 0.41 No irq handler for vector [12069.118900] do_IRQ: 0.41 No irq handler for vector [12069.122018] do_IRQ: 0.41 No irq handler for vector [12069.124728] do_IRQ: 0.41 No irq handler for vector [12069.127492] do_IRQ: 0.41 No irq handler for vector [12069.130227] do_IRQ: 0.41 No irq handler for vector [12089.860933] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 03:13:18 (1790752398) [12089.877327] do_IRQ: 1 callbacks suppressed [12089.877336] do_IRQ: 0.41 No irq handler for vector [12089.889602] do_IRQ: 0.41 No irq handler for vector [12089.896146] do_IRQ: 0.41 No irq handler for vector [12089.901824] do_IRQ: 0.41 No irq handler for vector [12089.906375] do_IRQ: 0.41 No irq handler for vector [12089.914091] do_IRQ: 0.41 No irq handler for vector [12089.919371] do_IRQ: 0.41 No irq handler for vector [12089.925681] do_IRQ: 0.41 No irq handler for vector [12089.927717] do_IRQ: 0.41 No irq handler for vector [12089.933259] do_IRQ: 0.41 No irq handler for vector [12101.323889] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 03:13:30 (1790752410) [12101.345884] do_IRQ: 1 callbacks suppressed [12101.345895] do_IRQ: 0.41 No irq handler for vector [12101.350750] do_IRQ: 0.41 No irq handler for vector [12101.355788] do_IRQ: 0.41 No irq handler for vector [12101.360789] do_IRQ: 0.41 No irq handler for vector [12101.363332] do_IRQ: 0.41 No irq handler for vector [12101.371482] do_IRQ: 0.41 No irq handler for vector [12101.375474] do_IRQ: 0.41 No irq handler for vector [12101.380852] do_IRQ: 0.41 No irq handler for vector [12101.383713] do_IRQ: 0.41 No irq handler for vector [12101.389333] do_IRQ: 0.41 No irq handler for vector [12127.660768] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 03:13:56 (1790752436) [12127.678115] do_IRQ: 2 callbacks suppressed [12127.678123] do_IRQ: 0.41 No irq handler for vector [12127.683515] do_IRQ: 0.41 No irq handler for vector [12127.689608] do_IRQ: 0.41 No irq handler for vector [12127.695108] do_IRQ: 0.41 No irq handler for vector [12127.697579] do_IRQ: 0.41 No irq handler for vector [12127.701110] do_IRQ: 0.41 No irq handler for vector [12127.703975] do_IRQ: 0.41 No irq handler for vector [12127.708323] do_IRQ: 0.41 No irq handler for vector [12127.711811] do_IRQ: 0.41 No irq handler for vector [12127.714580] do_IRQ: 0.41 No irq handler for vector [12156.174513] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 03:14:25 (1790752465) [12156.184472] do_IRQ: 1 callbacks suppressed [12156.184483] do_IRQ: 0.41 No irq handler for vector [12156.186505] do_IRQ: 0.41 No irq handler for vector [12156.191508] do_IRQ: 0.41 No irq handler for vector [12156.196619] do_IRQ: 0.41 No irq handler for vector [12156.204682] do_IRQ: 0.41 No irq handler for vector [12156.210505] do_IRQ: 0.41 No irq handler for vector [12156.216050] do_IRQ: 0.41 No irq handler for vector [12156.218258] do_IRQ: 0.41 No irq handler for vector [12156.222397] do_IRQ: 0.41 No irq handler for vector [12156.225148] do_IRQ: 0.41 No irq handler for vector [12167.008551] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 03:14:36 (1790752476) [12167.017427] do_IRQ: 2 callbacks suppressed [12167.017437] do_IRQ: 0.41 No irq handler for vector [12167.019497] do_IRQ: 0.41 No irq handler for vector [12167.022294] do_IRQ: 0.41 No irq handler for vector [12167.025029] do_IRQ: 0.41 No irq handler for vector [12167.027224] do_IRQ: 0.41 No irq handler for vector [12167.029441] do_IRQ: 0.41 No irq handler for vector [12167.032518] do_IRQ: 0.41 No irq handler for vector [12167.035489] do_IRQ: 0.41 No irq handler for vector [12167.037658] do_IRQ: 0.41 No irq handler for vector [12167.040348] do_IRQ: 0.41 No irq handler for vector [12187.618360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12187.627341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12187.627655] do_IRQ: 2 callbacks suppressed [12187.627660] do_IRQ: 0.41 No irq handler for vector [12187.656867] do_IRQ: 0.41 No irq handler for vector [12187.660038] do_IRQ: 0.41 No irq handler for vector [12187.665047] do_IRQ: 0.41 No irq handler for vector [12187.669195] do_IRQ: 0.41 No irq handler for vector [12187.672980] do_IRQ: 0.41 No irq handler for vector [12187.678332] do_IRQ: 0.41 No irq handler for vector [12187.682397] do_IRQ: 0.41 No irq handler for vector [12187.687099] do_IRQ: 0.41 No irq handler for vector [12187.692005] do_IRQ: 0.41 No irq handler for vector [12187.701782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12190.094253] Lustre: server umount lustre-MDT0000 complete [12193.252308] LustreError: 8440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12193.268825] do_IRQ: 3 callbacks suppressed [12193.268833] do_IRQ: 0.41 No irq handler for vector [12193.275069] do_IRQ: 0.41 No irq handler for vector [12193.277318] do_IRQ: 0.41 No irq handler for vector [12193.283326] do_IRQ: 0.41 No irq handler for vector [12193.286727] do_IRQ: 0.41 No irq handler for vector [12193.290120] do_IRQ: 0.41 No irq handler for vector [12193.294711] do_IRQ: 0.41 No irq handler for vector [12193.297799] do_IRQ: 0.41 No irq handler for vector [12193.302282] do_IRQ: 0.41 No irq handler for vector [12193.306706] do_IRQ: 0.41 No irq handler for vector [12193.313805] LustreError: 8440:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12196.427924] LustreError: 61189:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790752507 with bad export cookie 13675400811988760641 [12196.431367] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12196.436579] LustreError: 61189:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [12196.763026] Lustre: server umount lustre-MDT0001 complete [12213.480921] Lustre: server umount lustre-OST0000 complete [12213.484105] do_IRQ: 7 callbacks suppressed [12213.484116] do_IRQ: 0.41 No irq handler for vector [12213.498736] do_IRQ: 0.41 No irq handler for vector [12213.504965] do_IRQ: 0.41 No irq handler for vector [12213.514294] do_IRQ: 0.41 No irq handler for vector [12213.517700] do_IRQ: 0.41 No irq handler for vector [12213.520268] do_IRQ: 0.41 No irq handler for vector [12213.523190] do_IRQ: 0.41 No irq handler for vector [12213.528378] do_IRQ: 0.41 No irq handler for vector [12213.531296] do_IRQ: 0.41 No irq handler for vector [12213.538214] do_IRQ: 0.41 No irq handler for vector [12214.752226] Lustre: 3639:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790752509/real 1790752509] req@ffff8eb0336ab800 x1877727469267072/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790752525 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12214.763813] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12214.769186] Lustre: Skipped 3 previous similar messages [12217.574319] Lustre: 3641:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790752512/real 1790752512] req@ffff8eb004ecea00 x1877727469267456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790752528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12219.873581] Lustre: 3639:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790752514/real 1790752514] req@ffff8eb004ecd500 x1877727469267712/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790752530 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12219.913901] do_IRQ: 5 callbacks suppressed [12219.913908] do_IRQ: 0.41 No irq handler for vector [12219.927114] do_IRQ: 0.41 No irq handler for vector [12219.934012] do_IRQ: 0.41 No irq handler for vector [12219.941101] do_IRQ: 0.41 No irq handler for vector [12219.954591] do_IRQ: 0.41 No irq handler for vector [12219.956658] do_IRQ: 0.41 No irq handler for vector [12219.969439] do_IRQ: 0.41 No irq handler for vector [12219.972929] do_IRQ: 0.41 No irq handler for vector [12219.977209] do_IRQ: 0.41 No irq handler for vector [12219.981373] do_IRQ: 0.41 No irq handler for vector [12221.684179] Lustre: server umount lustre-OST0001 complete [12236.321636] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [12236.326736] do_IRQ: 2 callbacks suppressed [12236.326748] do_IRQ: 0.41 No irq handler for vector [12236.329417] do_IRQ: 0.41 No irq handler for vector [12236.341018] do_IRQ: 0.41 No irq handler for vector [12236.349938] do_IRQ: 0.41 No irq handler for vector [12236.352512] do_IRQ: 0.41 No irq handler for vector [12236.355512] do_IRQ: 0.41 No irq handler for vector [12236.358591] do_IRQ: 0.41 No irq handler for vector [12236.361530] do_IRQ: 0.41 No irq handler for vector [12236.367457] do_IRQ: 0.41 No irq handler for vector [12236.372839] do_IRQ: 0.41 No irq handler for vector [12239.446472] Key type lgssc unregistered [12239.754718] LNet: 108219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12239.761979] LNetError: 108219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12239.773388] LNet: Removed LNI 192.168.206.129@tcp [12240.822153] Key type .llcrypt unregistered [12240.825124] Key type ._llcrypt unregistered [12258.422409] Key type ._llcrypt registered [12258.424629] do_IRQ: 7 callbacks suppressed [12258.424637] do_IRQ: 0.41 No irq handler for vector [12258.425928] Key type .llcrypt registered [12258.433533] do_IRQ: 0.41 No irq handler for vector [12258.437146] do_IRQ: 0.41 No irq handler for vector [12258.440616] do_IRQ: 0.41 No irq handler for vector [12258.443311] do_IRQ: 0.41 No irq handler for vector [12258.449411] do_IRQ: 0.41 No irq handler for vector [12258.453374] do_IRQ: 0.41 No irq handler for vector [12258.455448] do_IRQ: 0.41 No irq handler for vector [12258.459879] do_IRQ: 0.41 No irq handler for vector [12258.461359] do_IRQ: 0.41 No irq handler for vector [12258.557476] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [12259.496924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12259.550403] alg: No test for adler32 (adler32-zlib) [12260.544939] Lustre: Lustre: Build Version: 2.17.59_3_g21e240e [12260.762690] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [12262.400212] Key type lgssc registered [12263.418977] Lustre: Echo OBD driver; http://www.lustre.org/ [12263.427423] do_IRQ: 7 callbacks suppressed [12263.427433] do_IRQ: 0.41 No irq handler for vector [12263.433444] do_IRQ: 0.41 No irq handler for vector [12263.439250] do_IRQ: 0.41 No irq handler for vector [12263.443044] do_IRQ: 0.41 No irq handler for vector [12263.449045] do_IRQ: 0.41 No irq handler for vector [12263.452134] do_IRQ: 0.41 No irq handler for vector [12263.455010] do_IRQ: 0.41 No irq handler for vector [12263.461041] do_IRQ: 0.41 No irq handler for vector [12263.465239] do_IRQ: 0.41 No irq handler for vector [12263.468379] do_IRQ: 0.41 No irq handler for vector [12278.238653] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [12278.246063] do_IRQ: 1 callbacks suppressed [12278.246071] do_IRQ: 0.41 No irq handler for vector [12278.251295] do_IRQ: 0.41 No irq handler for vector [12278.259730] do_IRQ: 0.41 No irq handler for vector [12278.262180] do_IRQ: 0.41 No irq handler for vector [12278.267947] do_IRQ: 0.41 No irq handler for vector [12278.270542] do_IRQ: 0.41 No irq handler for vector [12278.274061] do_IRQ: 0.41 No irq handler for vector [12278.276721] do_IRQ: 0.41 No irq handler for vector [12278.279256] do_IRQ: 0.41 No irq handler for vector [12278.287211] do_IRQ: 0.41 No irq handler for vector [12289.251835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12289.257920] do_IRQ: 1 callbacks suppressed [12289.257930] do_IRQ: 0.41 No irq handler for vector [12289.259697] do_IRQ: 0.41 No irq handler for vector [12289.262644] do_IRQ: 0.41 No irq handler for vector [12289.265466] do_IRQ: 0.41 No irq handler for vector [12289.269619] do_IRQ: 0.41 No irq handler for vector [12289.272673] do_IRQ: 0.41 No irq handler for vector [12289.274917] do_IRQ: 0.41 No irq handler for vector [12289.278472] do_IRQ: 0.41 No irq handler for vector [12289.280783] do_IRQ: 0.41 No irq handler for vector [12289.280811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12289.282664] do_IRQ: 0.41 No irq handler for vector [12290.610949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12294.662409] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [12294.667656] do_IRQ: 3 callbacks suppressed [12294.667665] do_IRQ: 0.41 No irq handler for vector [12294.682999] do_IRQ: 0.41 No irq handler for vector [12294.692726] do_IRQ: 0.41 No irq handler for vector [12294.700293] do_IRQ: 0.41 No irq handler for vector [12294.703954] do_IRQ: 0.41 No irq handler for vector [12294.710273] do_IRQ: 0.41 No irq handler for vector [12294.716131] do_IRQ: 0.41 No irq handler for vector [12294.724433] do_IRQ: 0.41 No irq handler for vector [12294.726933] do_IRQ: 0.41 No irq handler for vector [12294.731778] do_IRQ: 0.41 No irq handler for vector [12300.791491] LustreError: 110310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12300.814534] LustreError: 110310:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12300.816711] do_IRQ: 1 callbacks suppressed [12300.816719] do_IRQ: 0.41 No irq handler for vector [12300.832728] do_IRQ: 0.41 No irq handler for vector [12300.839979] do_IRQ: 0.41 No irq handler for vector [12300.845180] do_IRQ: 0.41 No irq handler for vector [12300.846424] do_IRQ: 0.41 No irq handler for vector [12300.851501] do_IRQ: 0.41 No irq handler for vector [12300.859503] do_IRQ: 0.41 No irq handler for vector [12300.863877] do_IRQ: 0.41 No irq handler for vector [12300.869775] do_IRQ: 0.41 No irq handler for vector [12300.871583] do_IRQ: 0.41 No irq handler for vector [12304.251202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12304.762233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12308.863485] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [12308.873128] do_IRQ: 3 callbacks suppressed [12308.873136] do_IRQ: 0.41 No irq handler for vector [12308.884336] do_IRQ: 0.41 No irq handler for vector [12308.892820] do_IRQ: 0.41 No irq handler for vector [12308.897223] do_IRQ: 0.41 No irq handler for vector [12308.899914] do_IRQ: 0.41 No irq handler for vector [12308.917558] do_IRQ: 0.41 No irq handler for vector [12308.920307] do_IRQ: 0.41 No irq handler for vector [12308.923540] do_IRQ: 0.41 No irq handler for vector [12308.930276] do_IRQ: 0.41 No irq handler for vector [12308.934511] do_IRQ: 0.41 No irq handler for vector [12312.079650] Lustre: 111607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12320.583362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12320.588816] do_IRQ: 2 callbacks suppressed [12320.588824] do_IRQ: 0.41 No irq handler for vector [12320.595738] do_IRQ: 0.41 No irq handler for vector [12320.597569] do_IRQ: 0.41 No irq handler for vector [12320.600669] do_IRQ: 0.41 No irq handler for vector [12320.602681] do_IRQ: 0.41 No irq handler for vector [12320.607180] do_IRQ: 0.41 No irq handler for vector [12320.608842] do_IRQ: 0.41 No irq handler for vector [12320.613129] do_IRQ: 0.41 No irq handler for vector [12320.621075] do_IRQ: 0.41 No irq handler for vector [12320.622791] do_IRQ: 0.41 No irq handler for vector [12320.944183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12326.101656] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [12326.106094] do_IRQ: 2 callbacks suppressed [12326.106104] do_IRQ: 0.41 No irq handler for vector [12326.109431] do_IRQ: 0.41 No irq handler for vector [12326.113833] do_IRQ: 0.41 No irq handler for vector [12326.118547] do_IRQ: 0.41 No irq handler for vector [12326.122497] do_IRQ: 0.41 No irq handler for vector [12326.127667] do_IRQ: 0.41 No irq handler for vector [12326.130656] do_IRQ: 0.41 No irq handler for vector [12326.133935] do_IRQ: 0.41 No irq handler for vector [12326.137375] do_IRQ: 0.41 No irq handler for vector [12326.141808] do_IRQ: 0.41 No irq handler for vector [12330.170930] LustreError: 112115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12330.188068] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:12213 to 0x280000400:12257) [12335.596101] LustreError: 112115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12335.634280] do_IRQ: 4 callbacks suppressed [12335.634286] do_IRQ: 0.41 No irq handler for vector [12335.641148] do_IRQ: 0.41 No irq handler for vector [12335.650665] do_IRQ: 0.41 No irq handler for vector [12335.657868] do_IRQ: 0.41 No irq handler for vector [12335.673698] do_IRQ: 0.41 No irq handler for vector [12335.680116] do_IRQ: 0.41 No irq handler for vector [12335.688143] do_IRQ: 0.41 No irq handler for vector [12335.695840] do_IRQ: 0.41 No irq handler for vector [12335.704169] do_IRQ: 0.41 No irq handler for vector [12335.706870] do_IRQ: 0.41 No irq handler for vector [12336.447342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12336.645798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:29425 to 0x280000403:29505) [12336.911760] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12342.265986] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:12045 to 0x2c0000400:12065) [12342.279208] do_IRQ: 4 callbacks suppressed [12342.279225] do_IRQ: 0.41 No irq handler for vector [12342.280272] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23089 to 0x2c0000403:23169) [12342.282521] do_IRQ: 0.41 No irq handler for vector [12342.292789] do_IRQ: 0.41 No irq handler for vector [12342.310562] do_IRQ: 0.41 No irq handler for vector [12342.316604] do_IRQ: 0.41 No irq handler for vector [12342.319972] do_IRQ: 0.41 No irq handler for vector [12342.321252] do_IRQ: 0.41 No irq handler for vector [12342.328198] do_IRQ: 0.41 No irq handler for vector [12342.335454] do_IRQ: 0.41 No irq handler for vector [12342.338464] do_IRQ: 0.41 No irq handler for vector [12343.473170] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [12351.417664] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12351.421439] do_IRQ: 2 callbacks suppressed [12351.421450] do_IRQ: 0.41 No irq handler for vector [12351.426619] do_IRQ: 0.41 No irq handler for vector [12351.428387] do_IRQ: 0.41 No irq handler for vector [12351.430901] do_IRQ: 0.41 No irq handler for vector [12351.432371] do_IRQ: 0.41 No irq handler for vector [12351.437304] do_IRQ: 0.41 No irq handler for vector [12351.439754] do_IRQ: 0.41 No irq handler for vector [12351.449703] do_IRQ: 0.41 No irq handler for vector [12351.463371] do_IRQ: 0.41 No irq handler for vector [12351.464984] do_IRQ: 0.41 No irq handler for vector [12354.791286] Lustre: 113790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12362.953679] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 03:17:52 (1790752672) [12362.963853] do_IRQ: 2 callbacks suppressed [12362.963863] do_IRQ: 0.41 No irq handler for vector [12362.973735] do_IRQ: 0.41 No irq handler for vector [12362.976450] do_IRQ: 0.41 No irq handler for vector [12362.978687] do_IRQ: 0.41 No irq handler for vector [12362.980983] do_IRQ: 0.41 No irq handler for vector [12362.983922] do_IRQ: 0.41 No irq handler for vector [12362.987250] do_IRQ: 0.41 No irq handler for vector [12362.989653] do_IRQ: 0.41 No irq handler for vector [12362.992339] do_IRQ: 0.41 No irq handler for vector [12362.994339] do_IRQ: 0.41 No irq handler for vector [12370.056917] LNet: 114282:0:(debug.c:375:cfs_str2mask()) unknown mask ''. [12370.056917] mask usage: [+|-] ... [12370.076125] do_IRQ: 1 callbacks suppressed [12370.076136] do_IRQ: 0.41 No irq handler for vector [12370.078056] do_IRQ: 0.41 No irq handler for vector [12370.080104] do_IRQ: 0.41 No irq handler for vector [12370.083658] do_IRQ: 0.41 No irq handler for vector [12370.091328] do_IRQ: 0.41 No irq handler for vector [12370.104094] do_IRQ: 0.41 No irq handler for vector [12370.111446] do_IRQ: 0.41 No irq handler for vector [12370.119954] do_IRQ: 0.41 No irq handler for vector [12370.132842] do_IRQ: 0.41 No irq handler for vector [12370.143496] do_IRQ: 0.41 No irq handler for vector [12370.772651] LNet: 114326:0:(debug.c:375:cfs_str2mask()) unknown mask ''. [12370.772651] mask usage: [+|-] ... [12370.787137] LNet: 114326:0:(debug.c:375:cfs_str2mask()) Skipped 3 previous similar messages [12370.863934] Lustre: DEBUG MARKER:  [12370.867830] Lustre: DEBUG MARKER:  [12371.811964] LNet: 114428:0:(debug.c:375:cfs_str2mask()) unknown mask ''. [12371.811964] mask usage: [+|-] ... [12371.820542] LNet: 114428:0:(debug.c:375:cfs_str2mask()) Skipped 1 previous similar message [12428.456501] LNet: 115991:0:(debug.c:375:cfs_str2mask()) unknown mask ''. [12428.456501] mask usage: [+|-] ... [12428.469473] do_IRQ: 8 callbacks suppressed [12428.469484] do_IRQ: 0.41 No irq handler for vector [12428.469486] LNet: 115991:0:(debug.c:375:cfs_str2mask()) Skipped 1 previous similar message [12428.472258] do_IRQ: 0.41 No irq handler for vector [12428.476741] do_IRQ: 0.41 No irq handler for vector [12428.486614] do_IRQ: 0.41 No irq handler for vector [12428.490259] do_IRQ: 0.41 No irq handler for vector [12428.493230] do_IRQ: 0.41 No irq handler for vector [12428.496033] do_IRQ: 0.41 No irq handler for vector [12428.498954] do_IRQ: 0.41 No irq handler for vector [12428.501501] do_IRQ: 0.41 No irq handler for vector [12428.504054] do_IRQ: 0.41 No irq handler for vector [12429.190672] Lustre: DEBUG MARKER:  [12429.198504] Lustre: DEBUG MARKER:  [12485.608254] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12485.618260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12485.626035] Lustre: Skipped 2 previous similar messages [12485.636201] do_IRQ: 3 callbacks suppressed [12485.636208] do_IRQ: 0.41 No irq handler for vector [12485.647225] do_IRQ: 0.41 No irq handler for vector [12485.649898] do_IRQ: 0.41 No irq handler for vector [12485.662075] do_IRQ: 0.41 No irq handler for vector [12485.665686] do_IRQ: 0.41 No irq handler for vector [12485.668310] do_IRQ: 0.41 No irq handler for vector [12485.674448] do_IRQ: 0.41 No irq handler for vector [12485.676148] do_IRQ: 0.41 No irq handler for vector [12485.678175] do_IRQ: 0.41 No irq handler for vector [12485.680836] do_IRQ: 0.41 No irq handler for vector [12485.691584] Lustre: Skipped 3 previous similar messages [12490.714148] Lustre: server umount lustre-MDT0000 complete [12490.723149] do_IRQ: 5 callbacks suppressed [12490.723166] do_IRQ: 0.41 No irq handler for vector [12490.737732] do_IRQ: 0.41 No irq handler for vector [12490.739361] do_IRQ: 0.41 No irq handler for vector [12490.752762] do_IRQ: 0.41 No irq handler for vector [12490.760666] do_IRQ: 0.41 No irq handler for vector [12490.766562] do_IRQ: 0.41 No irq handler for vector [12490.771831] do_IRQ: 0.41 No irq handler for vector [12490.781559] do_IRQ: 0.41 No irq handler for vector [12490.789307] do_IRQ: 0.41 No irq handler for vector [12490.790765] do_IRQ: 0.41 No irq handler for vector [12490.798863] LustreError: 110309:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12490.825945] LustreError: 110309:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12495.842370] LustreError: 110310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12495.869155] do_IRQ: 3 callbacks suppressed [12495.869166] do_IRQ: 0.41 No irq handler for vector [12495.874740] do_IRQ: 0.41 No irq handler for vector [12495.877459] do_IRQ: 0.41 No irq handler for vector [12495.883911] do_IRQ: 0.41 No irq handler for vector [12495.891684] do_IRQ: 0.41 No irq handler for vector [12495.894101] do_IRQ: 0.41 No irq handler for vector [12495.897968] do_IRQ: 0.41 No irq handler for vector [12495.900657] do_IRQ: 0.41 No irq handler for vector [12495.904233] do_IRQ: 0.41 No irq handler for vector [12495.906891] do_IRQ: 0.41 No irq handler for vector [12495.913633] LustreError: 110310:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12498.773560] LustreError: 110289:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790752809 with bad export cookie 7524679518676990003 [12498.782837] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12498.784495] LustreError: 110289:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 2 previous similar messages [12499.267435] Lustre: server umount lustre-MDT0001 complete [12516.128204] Lustre: 108697:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790752811/real 1790752811] req@ffff8eb00e194e00 x1877740168007936/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790752827 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12516.168190] do_IRQ: 7 callbacks suppressed [12516.168201] do_IRQ: 0.41 No irq handler for vector [12516.173790] do_IRQ: 0.41 No irq handler for vector [12516.174130] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12516.174134] Lustre: Skipped 1 previous similar message [12516.194418] do_IRQ: 0.41 No irq handler for vector [12516.197118] do_IRQ: 0.41 No irq handler for vector [12516.199614] do_IRQ: 0.41 No irq handler for vector [12516.202463] do_IRQ: 0.41 No irq handler for vector [12516.210445] do_IRQ: 0.41 No irq handler for vector [12516.219050] do_IRQ: 0.41 No irq handler for vector [12516.224606] do_IRQ: 0.41 No irq handler for vector [12516.230052] do_IRQ: 0.41 No irq handler for vector [12518.505586] Lustre: server umount lustre-OST0000 complete [12519.402118] Lustre: 108699:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790752814/real 1790752814] req@ffff8eb1114a9180 x1877740168008192/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790752830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12521.313080] Lustre: 108697:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790752816/real 1790752816] req@ffff8eb0238bca80 x1877740168008448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790752832 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12521.361395] do_IRQ: 3 callbacks suppressed [12521.361401] do_IRQ: 0.41 No irq handler for vector [12521.374802] do_IRQ: 0.41 No irq handler for vector [12521.382100] do_IRQ: 0.41 No irq handler for vector [12521.387675] do_IRQ: 0.41 No irq handler for vector [12521.397039] do_IRQ: 0.41 No irq handler for vector [12521.398744] do_IRQ: 0.41 No irq handler for vector [12521.404463] do_IRQ: 0.41 No irq handler for vector [12521.413483] do_IRQ: 0.41 No irq handler for vector [12521.419137] do_IRQ: 0.41 No irq handler for vector [12521.423718] do_IRQ: 0.41 No irq handler for vector [12524.576383] Lustre: 108699:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790752819/real 1790752819] req@ffff8eb00e194700 x1877740168008704/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790752835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12526.126484] Lustre: server umount lustre-OST0001 complete [12541.977604] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [12541.985454] do_IRQ: 3 callbacks suppressed [12541.985462] do_IRQ: 0.41 No irq handler for vector [12541.993745] do_IRQ: 0.41 No irq handler for vector [12541.996733] do_IRQ: 0.41 No irq handler for vector [12542.001100] do_IRQ: 0.41 No irq handler for vector [12542.004432] do_IRQ: 0.41 No irq handler for vector [12542.007616] do_IRQ: 0.41 No irq handler for vector [12542.010975] do_IRQ: 0.41 No irq handler for vector [12542.014121] do_IRQ: 0.41 No irq handler for vector [12542.019412] do_IRQ: 0.41 No irq handler for vector [12542.022636] do_IRQ: 0.41 No irq handler for vector [12544.940911] Key type lgssc unregistered [12545.213845] LNet: 119801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12545.226801] LNetError: 119801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12545.249612] LNet: Removed LNI 192.168.206.129@tcp [12546.171151] Key type .llcrypt unregistered [12546.175894] Key type ._llcrypt unregistered [12560.323631] Key type ._llcrypt registered [12560.326803] do_IRQ: 7 callbacks suppressed [12560.326812] do_IRQ: 0.41 No irq handler for vector [12560.329651] do_IRQ: 0.41 No irq handler for vector [12560.334323] do_IRQ: 0.41 No irq handler for vector [12560.337226] do_IRQ: 0.41 No irq handler for vector [12560.340322] do_IRQ: 0.41 No irq handler for vector [12560.343304] do_IRQ: 0.41 No irq handler for vector [12560.351070] do_IRQ: 0.41 No irq handler for vector [12560.354459] do_IRQ: 0.41 No irq handler for vector [12560.357321] do_IRQ: 0.41 No irq handler for vector [12560.361357] do_IRQ: 0.41 No irq handler for vector [12560.365785] Key type .llcrypt registered [12560.462161] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [12561.404324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12561.706492] alg: No test for adler32 (adler32-zlib) [12562.778754] Lustre: Lustre: Build Version: 2.17.59_3_g21e240e [12562.994815] LNet: Added LNI 192.168.206.129@tcp [8/256/0/180] [12564.687329] Key type lgssc registered [12566.240542] Lustre: Echo OBD driver; http://www.lustre.org/ [12566.245925] do_IRQ: 9 callbacks suppressed [12566.245934] do_IRQ: 0.41 No irq handler for vector [12566.248372] do_IRQ: 0.41 No irq handler for vector [12566.255181] do_IRQ: 0.41 No irq handler for vector [12566.262895] do_IRQ: 0.41 No irq handler for vector [12566.267665] do_IRQ: 0.41 No irq handler for vector [12566.276451] do_IRQ: 0.41 No irq handler for vector [12566.283284] do_IRQ: 0.41 No irq handler for vector [12566.294167] do_IRQ: 0.41 No irq handler for vector [12566.300144] do_IRQ: 0.41 No irq handler for vector [12566.302629] do_IRQ: 0.41 No irq handler for vector [12581.094764] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_modules_local [12581.099344] do_IRQ: 2 callbacks suppressed [12581.099356] do_IRQ: 0.41 No irq handler for vector [12581.105449] do_IRQ: 0.41 No irq handler for vector [12581.107788] do_IRQ: 0.41 No irq handler for vector [12581.112417] do_IRQ: 0.41 No irq handler for vector [12581.120514] do_IRQ: 0.41 No irq handler for vector [12581.123135] do_IRQ: 0.41 No irq handler for vector [12581.125799] do_IRQ: 0.41 No irq handler for vector [12581.128449] do_IRQ: 0.41 No irq handler for vector [12581.131159] do_IRQ: 0.41 No irq handler for vector [12581.141463] do_IRQ: 0.41 No irq handler for vector [12594.913333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12594.925180] do_IRQ: 1 callbacks suppressed [12594.925190] do_IRQ: 0.41 No irq handler for vector [12594.937115] do_IRQ: 0.41 No irq handler for vector [12594.943245] do_IRQ: 0.41 No irq handler for vector [12594.944571] do_IRQ: 0.41 No irq handler for vector [12594.950742] do_IRQ: 0.41 No irq handler for vector [12594.955811] do_IRQ: 0.41 No irq handler for vector [12594.964389] do_IRQ: 0.41 No irq handler for vector [12594.967148] do_IRQ: 0.41 No irq handler for vector [12594.969580] do_IRQ: 0.41 No irq handler for vector [12594.973690] do_IRQ: 0.41 No irq handler for vector [12594.981350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12596.386885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12600.311964] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [12600.315422] do_IRQ: 3 callbacks suppressed [12600.315434] do_IRQ: 0.41 No irq handler for vector [12600.324491] do_IRQ: 0.41 No irq handler for vector [12600.329117] do_IRQ: 0.41 No irq handler for vector [12600.332604] do_IRQ: 0.41 No irq handler for vector [12600.340003] do_IRQ: 0.41 No irq handler for vector [12600.343346] do_IRQ: 0.41 No irq handler for vector [12600.349516] do_IRQ: 0.41 No irq handler for vector [12600.353473] do_IRQ: 0.41 No irq handler for vector [12600.357465] do_IRQ: 0.41 No irq handler for vector [12600.361477] do_IRQ: 0.41 No irq handler for vector [12606.949061] LustreError: 121874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12606.975193] do_IRQ: 1 callbacks suppressed [12606.975200] do_IRQ: 0.41 No irq handler for vector [12606.982921] do_IRQ: 0.41 No irq handler for vector [12606.990851] do_IRQ: 0.41 No irq handler for vector [12606.993490] do_IRQ: 0.41 No irq handler for vector [12606.997516] do_IRQ: 0.41 No irq handler for vector [12606.999296] do_IRQ: 0.41 No irq handler for vector [12607.007387] do_IRQ: 0.41 No irq handler for vector [12607.016165] do_IRQ: 0.41 No irq handler for vector [12607.021263] do_IRQ: 0.41 No irq handler for vector [12607.024144] do_IRQ: 0.41 No irq handler for vector [12607.029648] LustreError: 121874:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12610.898818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12611.507646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12615.842187] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [12615.848671] do_IRQ: 5 callbacks suppressed [12615.848679] do_IRQ: 0.41 No irq handler for vector [12615.858715] do_IRQ: 0.41 No irq handler for vector [12615.863627] do_IRQ: 0.41 No irq handler for vector [12615.875103] do_IRQ: 0.41 No irq handler for vector [12615.879937] do_IRQ: 0.41 No irq handler for vector [12615.885845] do_IRQ: 0.41 No irq handler for vector [12615.890304] do_IRQ: 0.41 No irq handler for vector [12615.899075] do_IRQ: 0.41 No irq handler for vector [12615.903250] do_IRQ: 0.41 No irq handler for vector [12615.908580] do_IRQ: 0.41 No irq handler for vector [12618.598680] Lustre: 123170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12627.920510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12627.929197] do_IRQ: 2 callbacks suppressed [12627.929207] do_IRQ: 0.41 No irq handler for vector [12627.934580] do_IRQ: 0.41 No irq handler for vector [12627.941049] do_IRQ: 0.41 No irq handler for vector [12627.946641] do_IRQ: 0.41 No irq handler for vector [12627.952124] do_IRQ: 0.41 No irq handler for vector [12627.958487] do_IRQ: 0.41 No irq handler for vector [12627.962649] do_IRQ: 0.41 No irq handler for vector [12627.969462] do_IRQ: 0.41 No irq handler for vector [12627.978632] do_IRQ: 0.41 No irq handler for vector [12627.983588] do_IRQ: 0.41 No irq handler for vector [12628.405837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12634.302790] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [12634.307644] do_IRQ: 3 callbacks suppressed [12634.307654] do_IRQ: 0.41 No irq handler for vector [12634.309686] do_IRQ: 0.41 No irq handler for vector [12634.312368] do_IRQ: 0.41 No irq handler for vector [12634.314692] do_IRQ: 0.41 No irq handler for vector [12634.319584] do_IRQ: 0.41 No irq handler for vector [12634.323031] do_IRQ: 0.41 No irq handler for vector [12634.324787] do_IRQ: 0.41 No irq handler for vector [12634.326124] do_IRQ: 0.41 No irq handler for vector [12634.328551] do_IRQ: 0.41 No irq handler for vector [12634.330996] do_IRQ: 0.41 No irq handler for vector [12636.583421] LustreError: 123678:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12636.602171] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:12213 to 0x280000400:12289) [12641.717205] LustreError: 123679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12641.737626] LustreError: 123679:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12641.738514] do_IRQ: 4 callbacks suppressed [12641.738519] do_IRQ: 0.41 No irq handler for vector [12641.748883] do_IRQ: 0.41 No irq handler for vector [12641.751943] do_IRQ: 0.41 No irq handler for vector [12641.754979] do_IRQ: 0.41 No irq handler for vector [12641.758655] do_IRQ: 0.41 No irq handler for vector [12641.763012] do_IRQ: 0.41 No irq handler for vector [12641.765158] do_IRQ: 0.41 No irq handler for vector [12641.767576] do_IRQ: 0.41 No irq handler for vector [12641.770251] do_IRQ: 0.41 No irq handler for vector [12641.772550] do_IRQ: 0.41 No irq handler for vector [12641.775788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:29425 to 0x280000403:29537) [12644.732991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12645.167687] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12650.500849] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23089 to 0x2c0000403:23201) [12650.511907] do_IRQ: 4 callbacks suppressed [12650.511923] do_IRQ: 0.41 No irq handler for vector [12650.519627] do_IRQ: 0.41 No irq handler for vector [12650.523182] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:12045 to 0x2c0000400:12097) [12650.525497] do_IRQ: 0.41 No irq handler for vector [12650.553265] do_IRQ: 0.41 No irq handler for vector [12650.562616] do_IRQ: 0.41 No irq handler for vector [12650.572680] do_IRQ: 0.41 No irq handler for vector [12650.583893] do_IRQ: 0.41 No irq handler for vector [12650.588803] do_IRQ: 0.41 No irq handler for vector [12650.593745] do_IRQ: 0.41 No irq handler for vector [12650.600336] do_IRQ: 0.41 No irq handler for vector [12651.309146] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [12659.906158] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12659.909134] do_IRQ: 2 callbacks suppressed [12659.909145] do_IRQ: 0.41 No irq handler for vector [12659.917448] do_IRQ: 0.41 No irq handler for vector [12659.921820] do_IRQ: 0.41 No irq handler for vector [12659.923854] do_IRQ: 0.41 No irq handler for vector [12659.926688] do_IRQ: 0.41 No irq handler for vector [12659.928719] do_IRQ: 0.41 No irq handler for vector [12659.931491] do_IRQ: 0.41 No irq handler for vector [12659.935295] do_IRQ: 0.41 No irq handler for vector [12659.941876] do_IRQ: 0.41 No irq handler for vector [12659.946114] do_IRQ: 0.41 No irq handler for vector [12663.274980] Lustre: 125353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12670.210542] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 03:22:59 (1790752979) [12670.230138] do_IRQ: 3 callbacks suppressed [12670.230150] do_IRQ: 0.41 No irq handler for vector [12670.245398] do_IRQ: 0.41 No irq handler for vector [12670.255031] do_IRQ: 0.41 No irq handler for vector [12670.258033] do_IRQ: 0.41 No irq handler for vector [12670.262864] do_IRQ: 0.41 No irq handler for vector [12670.269741] do_IRQ: 0.41 No irq handler for vector [12670.273601] do_IRQ: 0.41 No irq handler for vector [12670.279209] do_IRQ: 0.41 No irq handler for vector [12670.283137] do_IRQ: 0.41 No irq handler for vector [12670.288113] do_IRQ: 0.41 No irq handler for vector [13687.182912] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 03:39:56 (1790753996) [13687.194556] do_IRQ: 1 callbacks suppressed [13687.194566] do_IRQ: 0.41 No irq handler for vector [13687.207382] do_IRQ: 0.41 No irq handler for vector [13687.215829] do_IRQ: 0.41 No irq handler for vector [13687.220944] do_IRQ: 0.41 No irq handler for vector [13687.229402] do_IRQ: 0.41 No irq handler for vector [13687.243773] do_IRQ: 0.41 No irq handler for vector [13687.247139] do_IRQ: 0.41 No irq handler for vector [13687.249747] do_IRQ: 0.41 No irq handler for vector [13687.252395] do_IRQ: 0.41 No irq handler for vector [13687.257791] do_IRQ: 0.41 No irq handler for vector [13702.961172] Lustre: *** cfs_fail_loc=327, val=500*** [13702.967890] do_IRQ: 1 callbacks suppressed [13702.967899] do_IRQ: 0.41 No irq handler for vector [13702.972297] do_IRQ: 0.41 No irq handler for vector [13702.976252] do_IRQ: 0.41 No irq handler for vector [13702.984271] do_IRQ: 0.41 No irq handler for vector [13702.990755] do_IRQ: 0.41 No irq handler for vector [13702.996196] do_IRQ: 0.41 No irq handler for vector [13702.998436] do_IRQ: 0.41 No irq handler for vector [13703.000920] do_IRQ: 0.41 No irq handler for vector [13703.004671] do_IRQ: 0.41 No irq handler for vector [13703.009200] do_IRQ: 0.41 No irq handler for vector [13703.875859] Lustre: *** cfs_fail_loc=327, val=500*** [13703.880689] Lustre: Skipped 512 previous similar messages [13735.212695] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 03:40:44 (1790754044) [13735.224066] do_IRQ: 4 callbacks suppressed [13735.224074] do_IRQ: 0.41 No irq handler for vector [13735.235131] do_IRQ: 0.41 No irq handler for vector [13735.237549] do_IRQ: 0.41 No irq handler for vector [13735.242849] do_IRQ: 0.41 No irq handler for vector [13735.247294] do_IRQ: 0.41 No irq handler for vector [13735.252162] do_IRQ: 0.41 No irq handler for vector [13735.256359] do_IRQ: 0.41 No irq handler for vector [13735.259709] do_IRQ: 0.41 No irq handler for vector [13735.262315] do_IRQ: 0.41 No irq handler for vector [13735.266348] do_IRQ: 0.41 No irq handler for vector [13740.296583] Lustre: *** cfs_fail_loc=328, val=500*** [13740.301520] Lustre: Skipped 512 previous similar messages [13740.303486] do_IRQ: 1 callbacks suppressed [13740.303497] do_IRQ: 0.41 No irq handler for vector [13740.322486] do_IRQ: 0.41 No irq handler for vector [13740.325763] do_IRQ: 0.41 No irq handler for vector [13740.329123] do_IRQ: 0.41 No irq handler for vector [13740.330906] do_IRQ: 0.41 No irq handler for vector [13740.337150] do_IRQ: 0.41 No irq handler for vector [13740.340779] do_IRQ: 0.41 No irq handler for vector [13740.346687] do_IRQ: 0.41 No irq handler for vector [13740.353144] do_IRQ: 0.41 No irq handler for vector [13740.356749] do_IRQ: 0.41 No irq handler for vector [13742.300270] Lustre: *** cfs_fail_loc=328, val=500*** [13742.303149] Lustre: Skipped 202 previous similar messages [13743.754996] Lustre: 166420:0:(ldlm_lockd.c:1329:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8eb136eb1180 x1877740472213248/t0(0) o101->b2a0235d-dd0b-4e2d-a6f1-f7d04c3a463e@192.168.206.29@tcp:520/0 lens 648/0 e 0 to 0 dl 1790754065 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [13744.836143] Lustre: 123701:0:(ldlm_lockd.c:1329:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8eb136eb3480 x1877740472214016/t0(0) o101->b2a0235d-dd0b-4e2d-a6f1-f7d04c3a463e@192.168.206.29@tcp:521/0 lens 648/0 e 0 to 0 dl 1790754066 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [13745.918383] Lustre: 121870:0:(ldlm_lockd.c:1329:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8eb034222300 x1877740472215168/t0(0) o101->b2a0235d-dd0b-4e2d-a6f1-f7d04c3a463e@192.168.206.29@tcp:522/0 lens 648/0 e 0 to 0 dl 1790754067 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [13745.935637] do_IRQ: 4 callbacks suppressed [13745.935646] do_IRQ: 0.41 No irq handler for vector [13745.940547] do_IRQ: 0.41 No irq handler for vector [13745.943463] do_IRQ: 0.41 No irq handler for vector [13745.946078] do_IRQ: 0.41 No irq handler for vector [13745.948469] do_IRQ: 0.41 No irq handler for vector [13745.951213] do_IRQ: 0.41 No irq handler for vector [13745.953584] do_IRQ: 0.41 No irq handler for vector [13745.955537] do_IRQ: 0.41 No irq handler for vector [13745.957284] do_IRQ: 0.41 No irq handler for vector [13745.960228] do_IRQ: 0.41 No irq handler for vector [13747.013296] Lustre: *** cfs_fail_loc=328, val=500*** [13747.016806] Lustre: Skipped 161 previous similar messages [13748.869814] Lustre: 121868:0:(ldlm_lockd.c:1329:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8eb0068cb800 x1877740472251520/t0(0) o101->b2a0235d-dd0b-4e2d-a6f1-f7d04c3a463e@192.168.206.29@tcp:525/0 lens 648/0 e 0 to 0 dl 1790754070 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [13748.891459] Lustre: 121868:0:(ldlm_lockd.c:1329:ldlm_handle_enqueue()) Skipped 1 previous similar message [13753.092883] Lustre: 125579:0:(ldlm_lockd.c:1329:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8eb01a44ea00 x1877740472254848/t0(0) o101->b2a0235d-dd0b-4e2d-a6f1-f7d04c3a463e@192.168.206.29@tcp:529/0 lens 648/0 e 0 to 0 dl 1790754074 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [13753.121086] do_IRQ: 5 callbacks suppressed [13753.121097] do_IRQ: 0.41 No irq handler for vector [13753.123190] Lustre: 125579:0:(ldlm_lockd.c:1329:ldlm_handle_enqueue()) Skipped 3 previous similar messages [13753.144250] do_IRQ: 0.41 No irq handler for vector [13753.147510] do_IRQ: 0.41 No irq handler for vector [13753.151149] do_IRQ: 0.41 No irq handler for vector [13753.154572] do_IRQ: 0.41 No irq handler for vector [13753.158325] do_IRQ: 0.41 No irq handler for vector [13753.161764] do_IRQ: 0.41 No irq handler for vector [13753.165952] do_IRQ: 0.41 No irq handler for vector [13753.170055] do_IRQ: 0.41 No irq handler for vector [13753.172958] do_IRQ: 0.41 No irq handler for vector [13755.518803] Lustre: *** cfs_fail_loc=328, val=500*** [13755.522104] Lustre: Skipped 105 previous similar messages [13761.212355] LustreError: 189357:0:(ldlm_lockd.c:3249:lock_reclaim_threshold_mb_store()) Failed to set lock_reclaim_threshold_mb, rc = -22. [13761.221788] do_IRQ: 3 callbacks suppressed [13761.221796] do_IRQ: 0.41 No irq handler for vector [13761.227474] do_IRQ: 0.41 No irq handler for vector [13761.232057] do_IRQ: 0.41 No irq handler for vector [13761.238351] do_IRQ: 0.41 No irq handler for vector [13761.242696] do_IRQ: 0.41 No irq handler for vector [13761.248449] do_IRQ: 0.41 No irq handler for vector [13761.251446] do_IRQ: 0.41 No irq handler for vector [13761.260711] do_IRQ: 0.41 No irq handler for vector [13761.266683] do_IRQ: 0.41 No irq handler for vector [13761.271978] do_IRQ: 0.41 No irq handler for vector [13775.032767] Lustre: DEBUG MARKER: == sanity test 134c: Lock memory accounting includes associated structures ========================================================== 03:41:24 (1790754084) [13775.046465] do_IRQ: 1 callbacks suppressed [13775.046476] do_IRQ: 0.41 No irq handler for vector [13775.057622] do_IRQ: 0.41 No irq handler for vector [13775.062960] do_IRQ: 0.41 No irq handler for vector [13775.067891] do_IRQ: 0.41 No irq handler for vector [13775.071268] do_IRQ: 0.41 No irq handler for vector [13775.074613] do_IRQ: 0.41 No irq handler for vector [13775.078504] do_IRQ: 0.41 No irq handler for vector [13775.081512] do_IRQ: 0.41 No irq handler for vector [13775.085185] do_IRQ: 0.41 No irq handler for vector [13775.088178] do_IRQ: 0.41 No irq handler for vector [13787.575933] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [13787.579352] do_IRQ: 1 callbacks suppressed [13787.579361] do_IRQ: 0.41 No irq handler for vector [13787.581583] do_IRQ: 0.41 No irq handler for vector [13787.584022] do_IRQ: 0.41 No irq handler for vector [13787.586221] do_IRQ: 0.41 No irq handler for vector [13787.588625] do_IRQ: 0.41 No irq handler for vector [13787.591158] do_IRQ: 0.41 No irq handler for vector [13787.593105] do_IRQ: 0.41 No irq handler for vector [13787.595576] do_IRQ: 0.41 No irq handler for vector [13787.597983] do_IRQ: 0.41 No irq handler for vector [13787.600017] do_IRQ: 0.41 No irq handler for vector [13788.927808] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [13790.457342] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 03:41:39 (1790754099) [13835.253505] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 03:42:24 (1790754144) [13835.261707] do_IRQ: 4 callbacks suppressed [13835.261717] do_IRQ: 0.41 No irq handler for vector [13835.265353] do_IRQ: 0.41 No irq handler for vector [13835.268789] do_IRQ: 0.41 No irq handler for vector [13835.274900] do_IRQ: 0.41 No irq handler for vector [13835.278603] do_IRQ: 0.41 No irq handler for vector [13835.280421] do_IRQ: 0.41 No irq handler for vector [13835.283148] do_IRQ: 0.41 No irq handler for vector [13835.285755] do_IRQ: 0.41 No irq handler for vector [13835.289685] do_IRQ: 0.41 No irq handler for vector [13835.293062] do_IRQ: 0.41 No irq handler for vector [13878.433246] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 03:43:07 (1790754187) [13878.445217] do_IRQ: 2 callbacks suppressed [13878.445230] do_IRQ: 0.41 No irq handler for vector [13878.454957] do_IRQ: 0.41 No irq handler for vector [13878.459503] do_IRQ: 0.41 No irq handler for vector [13878.462345] do_IRQ: 0.41 No irq handler for vector [13878.464587] do_IRQ: 0.41 No irq handler for vector [13878.467856] do_IRQ: 0.41 No irq handler for vector [13878.470691] do_IRQ: 0.41 No irq handler for vector [13878.475157] do_IRQ: 0.41 No irq handler for vector [13878.477755] do_IRQ: 0.41 No irq handler for vector [13878.480358] do_IRQ: 0.41 No irq handler for vector [13904.123568] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 03:43:33 (1790754213) [13904.138679] do_IRQ: 1 callbacks suppressed [13904.138687] do_IRQ: 0.41 No irq handler for vector [13904.145802] do_IRQ: 0.41 No irq handler for vector [13904.151798] do_IRQ: 0.41 No irq handler for vector [13904.157488] do_IRQ: 0.41 No irq handler for vector [13904.162669] do_IRQ: 0.41 No irq handler for vector [13904.168351] do_IRQ: 0.41 No irq handler for vector [13904.171588] do_IRQ: 0.41 No irq handler for vector [13904.174243] do_IRQ: 0.41 No irq handler for vector [13904.176763] do_IRQ: 0.41 No irq handler for vector [13904.184165] do_IRQ: 0.41 No irq handler for vector [13934.408137] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 03:44:03 (1790754243) [13934.414243] do_IRQ: 1 callbacks suppressed [13934.414254] do_IRQ: 0.41 No irq handler for vector [13934.422387] do_IRQ: 0.41 No irq handler for vector [13934.428861] do_IRQ: 0.41 No irq handler for vector [13934.430856] do_IRQ: 0.41 No irq handler for vector [13934.438120] do_IRQ: 0.41 No irq handler for vector [13934.443349] do_IRQ: 0.41 No irq handler for vector [13934.446658] do_IRQ: 0.41 No irq handler for vector [13934.450103] do_IRQ: 0.41 No irq handler for vector [13934.453177] do_IRQ: 0.41 No irq handler for vector [13934.456277] do_IRQ: 0.41 No irq handler for vector [13954.893846] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 03:44:24 (1790754264) [13954.903568] do_IRQ: 1 callbacks suppressed [13954.903579] do_IRQ: 0.41 No irq handler for vector [13954.918933] do_IRQ: 0.41 No irq handler for vector [13954.926976] do_IRQ: 0.41 No irq handler for vector [13954.933493] do_IRQ: 0.41 No irq handler for vector [13954.935993] do_IRQ: 0.41 No irq handler for vector [13954.938286] do_IRQ: 0.41 No irq handler for vector [13954.944213] do_IRQ: 0.41 No irq handler for vector [13954.952143] do_IRQ: 0.41 No irq handler for vector [13954.959766] do_IRQ: 0.41 No irq handler for vector [13954.965550] do_IRQ: 0.41 No irq handler for vector [13975.415192] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 03:44:44 (1790754284) [13975.425841] do_IRQ: 1 callbacks suppressed [13975.425851] do_IRQ: 0.41 No irq handler for vector [13975.435499] do_IRQ: 0.41 No irq handler for vector [13975.438498] do_IRQ: 0.41 No irq handler for vector [13975.443990] do_IRQ: 0.41 No irq handler for vector [13975.448182] do_IRQ: 0.41 No irq handler for vector [13975.451810] do_IRQ: 0.41 No irq handler for vector [13975.454926] do_IRQ: 0.41 No irq handler for vector [13975.456659] do_IRQ: 0.41 No irq handler for vector [13975.459513] do_IRQ: 0.41 No irq handler for vector [13975.462657] do_IRQ: 0.41 No irq handler for vector [14008.180596] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 03:45:17 (1790754317) [14008.204567] do_IRQ: 1 callbacks suppressed [14008.204573] do_IRQ: 0.41 No irq handler for vector [14008.218426] do_IRQ: 0.41 No irq handler for vector [14008.224949] do_IRQ: 0.41 No irq handler for vector [14008.233404] do_IRQ: 0.41 No irq handler for vector [14008.239257] do_IRQ: 0.41 No irq handler for vector [14008.242255] do_IRQ: 0.41 No irq handler for vector [14008.244246] do_IRQ: 0.41 No irq handler for vector [14008.249101] do_IRQ: 0.41 No irq handler for vector [14008.256610] do_IRQ: 0.41 No irq handler for vector [14008.260387] do_IRQ: 0.41 No irq handler for vector [14038.000245] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 03:45:47 (1790754347) [14038.007516] do_IRQ: 1 callbacks suppressed [14038.007527] do_IRQ: 0.41 No irq handler for vector [14038.016936] do_IRQ: 0.41 No irq handler for vector [14038.020108] do_IRQ: 0.41 No irq handler for vector [14038.021897] do_IRQ: 0.41 No irq handler for vector [14038.027136] do_IRQ: 0.41 No irq handler for vector [14038.029978] do_IRQ: 0.41 No irq handler for vector [14038.035934] do_IRQ: 0.41 No irq handler for vector [14038.038155] do_IRQ: 0.41 No irq handler for vector [14038.039732] do_IRQ: 0.41 No irq handler for vector [14038.042517] do_IRQ: 0.41 No irq handler for vector [14063.594262] Lustre: DEBUG MARKER: == sanity test 150h: Verify extend fallocate updates the file size ========================================================== 03:46:12 (1790754372) [14063.614423] do_IRQ: 1 callbacks suppressed [14063.614429] do_IRQ: 0.41 No irq handler for vector [14063.628751] do_IRQ: 0.41 No irq handler for vector [14063.639889] do_IRQ: 0.41 No irq handler for vector [14063.644656] do_IRQ: 0.41 No irq handler for vector [14063.649666] do_IRQ: 0.41 No irq handler for vector [14063.652787] do_IRQ: 0.41 No irq handler for vector [14063.657102] do_IRQ: 0.41 No irq handler for vector [14063.663482] do_IRQ: 0.41 No irq handler for vector [14063.673045] do_IRQ: 0.41 No irq handler for vector [14063.676672] do_IRQ: 0.41 No irq handler for vector [14071.252551] Lustre: DEBUG MARKER: == sanity test 150ia: Verify fallocate zero-range ZERO functionality ========================================================== 03:46:20 (1790754380) [14071.267234] do_IRQ: 1 callbacks suppressed [14071.267258] do_IRQ: 0.41 No irq handler for vector [14071.283768] do_IRQ: 0.41 No irq handler for vector [14071.288226] do_IRQ: 0.41 No irq handler for vector [14071.297529] do_IRQ: 0.41 No irq handler for vector [14071.300242] do_IRQ: 0.41 No irq handler for vector [14071.306589] do_IRQ: 0.41 No irq handler for vector [14071.308088] do_IRQ: 0.41 No irq handler for vector [14071.321520] do_IRQ: 0.41 No irq handler for vector [14071.332564] do_IRQ: 0.41 No irq handler for vector [14071.340264] do_IRQ: 0.41 No irq handler for vector [14093.442418] Lustre: DEBUG MARKER: == sanity test 150ib: Verify fallocate zero-range PREALLOC functionality ========================================================== 03:46:42 (1790754402) [14093.460017] do_IRQ: 1 callbacks suppressed [14093.460028] do_IRQ: 0.41 No irq handler for vector [14093.474126] do_IRQ: 0.41 No irq handler for vector [14093.481981] do_IRQ: 0.41 No irq handler for vector [14093.485120] do_IRQ: 0.41 No irq handler for vector [14093.488013] do_IRQ: 0.41 No irq handler for vector [14093.492104] do_IRQ: 0.41 No irq handler for vector [14093.498941] do_IRQ: 0.41 No irq handler for vector [14093.501591] do_IRQ: 0.41 No irq handler for vector [14093.507332] do_IRQ: 0.41 No irq handler for vector [14093.509171] do_IRQ: 0.41 No irq handler for vector [14114.236594] Lustre: DEBUG MARKER: == sanity test 150ic: Verify fallocate LARGE zero PREALLOC functionality ========================================================== 03:47:03 (1790754423) [14114.246898] do_IRQ: 1 callbacks suppressed [14114.246907] do_IRQ: 0.41 No irq handler for vector [14114.256827] do_IRQ: 0.41 No irq handler for vector [14114.261166] do_IRQ: 0.41 No irq handler for vector [14114.265384] do_IRQ: 0.41 No irq handler for vector [14114.272932] do_IRQ: 0.41 No irq handler for vector [14114.279301] do_IRQ: 0.41 No irq handler for vector [14114.285121] do_IRQ: 0.41 No irq handler for vector [14114.288228] do_IRQ: 0.41 No irq handler for vector [14114.291250] do_IRQ: 0.41 No irq handler for vector [14114.294149] do_IRQ: 0.41 No irq handler for vector [14115.824511] Lustre: DEBUG MARKER: SKIP: sanity test_150ic only check on DoM component [14117.268848] Lustre: DEBUG MARKER: == sanity test 150id: fallocate that fails must not leave a dirty page behind ========================================================== 03:47:06 (1790754426) [14122.970857] Lustre: *** cfs_fail_loc=2303, val=0*** [14122.974535] do_IRQ: 3 callbacks suppressed [14122.974548] do_IRQ: 0.41 No irq handler for vector [14122.976450] LustreError: 192187:0:(osd_handler.c:2361:osd_trans_stop()) lustre-OST0001: failed truncate process: rc = -5 [14123.000726] do_IRQ: 0.41 No irq handler for vector [14123.007548] do_IRQ: 0.41 No irq handler for vector [14123.018967] do_IRQ: 0.41 No irq handler for vector [14123.020895] do_IRQ: 0.41 No irq handler for vector [14123.027169] do_IRQ: 0.41 No irq handler for vector [14123.033243] do_IRQ: 0.41 No irq handler for vector [14123.035897] do_IRQ: 0.41 No irq handler for vector [14123.038333] do_IRQ: 0.41 No irq handler for vector [14123.040645] do_IRQ: 0.41 No irq handler for vector [14123.513901] Lustre: *** cfs_fail_loc=2303, val=0*** [14123.518468] Lustre: Skipped 7 previous similar messages [14123.526148] LustreError: 123683:0:(osd_handler.c:2361:osd_trans_stop()) lustre-OST0001: failed truncate process: rc = -5 [14123.532817] LustreError: 123683:0:(osd_handler.c:2361:osd_trans_stop()) Skipped 7 previous similar messages [14124.850371] Lustre: *** cfs_fail_loc=2303, val=0*** [14124.855640] Lustre: Skipped 7 previous similar messages [14124.858722] LustreError: 123685:0:(osd_handler.c:2361:osd_trans_stop()) lustre-OST0001: failed truncate process: rc = -5 [14124.863747] LustreError: 123685:0:(osd_handler.c:2361:osd_trans_stop()) Skipped 7 previous similar messages [14127.345438] Lustre: *** cfs_fail_loc=2303, val=0*** [14127.353092] Lustre: Skipped 18 previous similar messages [14127.361472] LustreError: 123683:0:(osd_handler.c:2361:osd_trans_stop()) lustre-OST0001: failed truncate process: rc = -5 [14127.373408] LustreError: 123683:0:(osd_handler.c:2361:osd_trans_stop()) Skipped 18 previous similar messages [14131.763114] Lustre: *** cfs_fail_loc=2303, val=0*** [14131.767238] do_IRQ: 13 callbacks suppressed [14131.767252] do_IRQ: 0.41 No irq handler for vector [14131.770519] Lustre: Skipped 42 previous similar messages [14131.770578] LustreError: 123683:0:(osd_handler.c:2361:osd_trans_stop()) lustre-OST0001: failed truncate process: rc = -5 [14131.770583] LustreError: 123683:0:(osd_handler.c:2361:osd_trans_stop()) Skipped 42 previous similar messages [14131.820672] do_IRQ: 0.41 No irq handler for vector [14131.827709] do_IRQ: 0.41 No irq handler for vector [14131.829907] do_IRQ: 0.41 No irq handler for vector [14131.832796] do_IRQ: 0.41 No irq handler for vector [14131.837305] do_IRQ: 0.41 No irq handler for vector [14131.839857] do_IRQ: 0.41 No irq handler for vector [14131.842645] do_IRQ: 0.41 No irq handler for vector [14131.845443] do_IRQ: 0.41 No irq handler for vector [14131.852440] do_IRQ: 0.41 No irq handler for vector [14139.858318] Lustre: *** cfs_fail_loc=2303, val=0*** [14139.861395] do_IRQ: 1 callbacks suppressed [14139.861406] do_IRQ: 0.41 No irq handler for vector [14139.864328] do_IRQ: 0.41 No irq handler for vector [14139.867686] do_IRQ: 0.41 No irq handler for vector [14139.875022] do_IRQ: 0.41 No irq handler for vector [14139.878337] do_IRQ: 0.41 No irq handler for vector [14139.882172] do_IRQ: 0.41 No irq handler for vector [14139.885977] do_IRQ: 0.41 No irq handler for vector [14139.890067] do_IRQ: 0.41 No irq handler for vector [14139.892344] do_IRQ: 0.41 No irq handler for vector [14139.894542] do_IRQ: 0.41 No irq handler for vector [14139.899123] Lustre: Skipped 54 previous similar messages [14139.903008] LustreError: 123684:0:(osd_handler.c:2361:osd_trans_stop()) lustre-OST0001: failed truncate process: rc = -5 [14139.914875] LustreError: 123684:0:(osd_handler.c:2361:osd_trans_stop()) Skipped 54 previous similar messages [14155.869770] Lustre: *** cfs_fail_loc=2303, val=0*** [14155.872585] do_IRQ: 5 callbacks suppressed [14155.872603] do_IRQ: 0.41 No irq handler for vector [14155.877581] Lustre: Skipped 114 previous similar messages [14155.897358] do_IRQ: 0.41 No irq handler for vector [14155.912741] do_IRQ: 0.41 No irq handler for vector [14155.924705] do_IRQ: 0.41 No irq handler for vector [14155.931714] do_IRQ: 0.41 No irq handler for vector [14155.937006] do_IRQ: 0.41 No irq handler for vector [14155.940046] do_IRQ: 0.41 No irq handler for vector [14155.948952] do_IRQ: 0.41 No irq handler for vector [14155.954067] do_IRQ: 0.41 No irq handler for vector [14155.957385] do_IRQ: 0.41 No irq handler for vector [14156.890354] LustreError: 123685:0:(osd_handler.c:2361:osd_trans_stop()) lustre-OST0001: failed truncate process: rc = -5 [14156.895453] LustreError: 123685:0:(osd_handler.c:2361:osd_trans_stop()) Skipped 115 previous similar messages [14187.919863] Lustre: *** cfs_fail_loc=2303, val=0*** [14187.925699] do_IRQ: 3 callbacks suppressed [14187.925710] do_IRQ: 0.41 No irq handler for vector [14187.929236] Lustre: Skipped 203 previous similar messages [14187.937684] do_IRQ: 0.41 No irq handler for vector [14187.943691] do_IRQ: 0.41 No irq handler for vector [14187.948714] do_IRQ: 0.41 No irq handler for vector [14187.961387] do_IRQ: 0.41 No irq handler for vector [14187.966144] do_IRQ: 0.41 No irq handler for vector [14187.974547] do_IRQ: 0.41 No irq handler for vector [14187.982274] do_IRQ: 0.41 No irq handler for vector [14187.988898] do_IRQ: 0.41 No irq handler for vector [14188.000771] do_IRQ: 0.41 No irq handler for vector [14189.232135] LustreError: 123684:0:(osd_handler.c:2361:osd_trans_stop()) lustre-OST0001: failed truncate process: rc = -5 [14189.238130] LustreError: 123684:0:(osd_handler.c:2361:osd_trans_stop()) Skipped 206 previous similar messages [14251.921568] Lustre: *** cfs_fail_loc=2303, val=0*** [14251.923109] do_IRQ: 3 callbacks suppressed [14251.923117] do_IRQ: 0.41 No irq handler for vector [14251.930450] do_IRQ: 0.41 No irq handler for vector [14251.937406] Lustre: Skipped 393 previous similar messages [14251.940253] do_IRQ: 0.41 No irq handler for vector [14251.951189] do_IRQ: 0.41 No irq handler for vector [14251.969683] do_IRQ: 0.41 No irq handler for vector [14251.996906] do_IRQ: 0.41 No irq handler for vector [14252.004195] do_IRQ: 0.41 No irq handler for vector [14252.012041] do_IRQ: 0.41 No irq handler for vector [14252.016439] do_IRQ: 0.41 No irq handler for vector [14252.020180] do_IRQ: 0.41 No irq handler for vector [14253.371802] LustreError: 123683:0:(osd_handler.c:2361:osd_trans_stop()) lustre-OST0001: failed truncate process: rc = -5 [14253.381870] LustreError: 123683:0:(osd_handler.c:2361:osd_trans_stop()) Skipped 391 previous similar messages [14380.761861] Lustre: *** cfs_fail_loc=2303, val=0*** [14380.764492] do_IRQ: 3 callbacks suppressed [14380.764503] do_IRQ: 0.41 No irq handler for vector [14380.765636] Lustre: Skipped 826 previous similar messages [14380.779531] do_IRQ: 0.41 No irq handler for vector [14380.785755] do_IRQ: 0.41 No irq handler for vector [14380.787899] do_IRQ: 0.41 No irq handler for vector [14380.789728] do_IRQ: 0.41 No irq handler for vector [14380.795543] do_IRQ: 0.41 No irq handler for vector [14380.798381] do_IRQ: 0.41 No irq handler for vector [14380.802073] do_IRQ: 0.41 No irq handler for vector [14380.808797] do_IRQ: 0.41 No irq handler for vector [14380.812324] do_IRQ: 0.41 No irq handler for vector [14381.480827] LustreError: 192187:0:(osd_handler.c:2361:osd_trans_stop()) lustre-OST0001: failed truncate process: rc = -5 [14381.487627] LustreError: 192187:0:(osd_handler.c:2361:osd_trans_stop()) Skipped 827 previous similar messages [14636.950662] Lustre: *** cfs_fail_loc=2303, val=0*** [14636.954671] do_IRQ: 3 callbacks suppressed [14636.954683] do_IRQ: 0.41 No irq handler for vector [14636.954907] Lustre: Skipped 1583 previous similar messages [14636.958523] do_IRQ: 0.41 No irq handler for vector [14636.965620] do_IRQ: 0.41 No irq handler for vector [14636.968144] do_IRQ: 0.41 No irq handler for vector [14636.970261] do_IRQ: 0.41 No irq handler for vector [14636.972646] do_IRQ: 0.41 No irq handler for vector [14636.980114] do_IRQ: 0.41 No irq handler for vector [14636.982626] do_IRQ: 0.41 No irq handler for vector [14636.986491] do_IRQ: 0.41 No irq handler for vector [14636.992170] do_IRQ: 0.41 No irq handler for vector [14637.775407] LustreError: 123685:0:(osd_handler.c:2361:osd_trans_stop()) lustre-OST0001: failed truncate process: rc = -5 [14637.803729] LustreError: 123685:0:(osd_handler.c:2361:osd_trans_stop()) Skipped 1587 previous similar messages [14779.749472] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 03:58:08 (1790755088) [14779.760626] do_IRQ: 3 callbacks suppressed [14779.760636] do_IRQ: 0.41 No irq handler for vector [14779.763386] do_IRQ: 0.41 No irq handler for vector [14779.768692] do_IRQ: 0.41 No irq handler for vector [14779.772201] do_IRQ: 0.41 No irq handler for vector [14779.778267] do_IRQ: 0.41 No irq handler for vector [14779.781563] do_IRQ: 0.41 No irq handler for vector [14779.786046] do_IRQ: 0.41 No irq handler for vector [14779.789448] do_IRQ: 0.41 No irq handler for vector [14779.795505] do_IRQ: 0.41 No irq handler for vector [14779.798845] do_IRQ: 0.41 No irq handler for vector [14796.218263] bash (200946): drop_caches: 1 [14796.226357] do_IRQ: 2 callbacks suppressed [14796.226367] do_IRQ: 0.41 No irq handler for vector [14796.228722] do_IRQ: 0.41 No irq handler for vector [14796.231379] do_IRQ: 0.41 No irq handler for vector [14796.233911] do_IRQ: 0.41 No irq handler for vector [14796.236390] do_IRQ: 0.41 No irq handler for vector [14796.240168] do_IRQ: 0.41 No irq handler for vector [14796.242671] do_IRQ: 0.41 No irq handler for vector [14796.247887] do_IRQ: 0.41 No irq handler for vector [14796.251493] do_IRQ: 0.41 No irq handler for vector [14796.258035] do_IRQ: 0.41 No irq handler for vector [14808.325930] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 03:58:37 (1790755117) [14808.346761] do_IRQ: 2 callbacks suppressed [14808.346769] do_IRQ: 0.41 No irq handler for vector [14808.358033] do_IRQ: 0.41 No irq handler for vector [14808.363661] do_IRQ: 0.41 No irq handler for vector [14808.366916] do_IRQ: 0.41 No irq handler for vector [14808.369437] do_IRQ: 0.41 No irq handler for vector [14808.372366] do_IRQ: 0.41 No irq handler for vector [14808.375722] do_IRQ: 0.41 No irq handler for vector [14808.378095] do_IRQ: 0.41 No irq handler for vector [14808.380770] do_IRQ: 0.41 No irq handler for vector [14808.383515] do_IRQ: 0.41 No irq handler for vector [14815.108942] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 03:58:44 (1790755124) [14815.117533] do_IRQ: 1 callbacks suppressed [14815.117543] do_IRQ: 0.41 No irq handler for vector [14815.126283] do_IRQ: 0.41 No irq handler for vector [14815.128871] do_IRQ: 0.41 No irq handler for vector [14815.131806] do_IRQ: 0.41 No irq handler for vector [14815.138261] do_IRQ: 0.41 No irq handler for vector [14815.143835] do_IRQ: 0.41 No irq handler for vector [14815.149888] do_IRQ: 0.41 No irq handler for vector [14815.153817] do_IRQ: 0.41 No irq handler for vector [14815.156151] do_IRQ: 0.41 No irq handler for vector [14815.158016] do_IRQ: 0.41 No irq handler for vector [14823.263817] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 03:58:52 (1790755132) [14823.278955] do_IRQ: 1 callbacks suppressed [14823.278965] do_IRQ: 0.41 No irq handler for vector [14823.299338] do_IRQ: 0.41 No irq handler for vector [14823.307500] do_IRQ: 0.41 No irq handler for vector [14823.312397] do_IRQ: 0.41 No irq handler for vector [14823.315905] do_IRQ: 0.41 No irq handler for vector [14823.321280] do_IRQ: 0.41 No irq handler for vector [14823.333703] do_IRQ: 0.41 No irq handler for vector [14823.341662] do_IRQ: 0.41 No irq handler for vector [14823.348973] do_IRQ: 0.41 No irq handler for vector [14823.360158] do_IRQ: 0.41 No irq handler for vector [14830.308386] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 03:58:59 (1790755139) [14830.313161] do_IRQ: 1 callbacks suppressed [14830.313171] do_IRQ: 0.41 No irq handler for vector [14830.320753] do_IRQ: 0.41 No irq handler for vector [14830.322945] do_IRQ: 0.41 No irq handler for vector [14830.326112] do_IRQ: 0.41 No irq handler for vector [14830.329922] do_IRQ: 0.41 No irq handler for vector [14830.332792] do_IRQ: 0.41 No irq handler for vector [14830.339619] do_IRQ: 0.41 No irq handler for vector [14830.346742] do_IRQ: 0.41 No irq handler for vector [14830.349994] do_IRQ: 0.41 No irq handler for vector [14830.352445] do_IRQ: 0.41 No irq handler for vector [14837.298929] Lustre: DEBUG MARKER: == sanity test 154C: lfs fid2path on OST FID ============= 03:59:06 (1790755146) [14837.306609] do_IRQ: 1 callbacks suppressed [14837.306627] do_IRQ: 0.41 No irq handler for vector [14837.313808] do_IRQ: 0.41 No irq handler for vector [14837.316460] do_IRQ: 0.41 No irq handler for vector [14837.322972] do_IRQ: 0.41 No irq handler for vector [14837.327743] do_IRQ: 0.41 No irq handler for vector [14837.330738] do_IRQ: 0.41 No irq handler for vector [14837.336488] do_IRQ: 0.41 No irq handler for vector [14837.339336] do_IRQ: 0.41 No irq handler for vector [14837.340774] do_IRQ: 0.41 No irq handler for vector [14837.343238] do_IRQ: 0.41 No irq handler for vector [14861.510566] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 03:59:30 (1790755170) [14861.515919] do_IRQ: 1 callbacks suppressed [14861.515930] do_IRQ: 0.41 No irq handler for vector [14861.530373] do_IRQ: 0.41 No irq handler for vector [14861.533515] do_IRQ: 0.41 No irq handler for vector [14861.538197] do_IRQ: 0.41 No irq handler for vector [14861.542053] do_IRQ: 0.41 No irq handler for vector [14861.546904] do_IRQ: 0.41 No irq handler for vector [14861.552758] do_IRQ: 0.41 No irq handler for vector [14861.555578] do_IRQ: 0.41 No irq handler for vector [14861.565582] do_IRQ: 0.41 No irq handler for vector [14861.568669] do_IRQ: 0.41 No irq handler for vector [14863.581598] LustreError: 121879:0:(fld_handler.c:268:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [14872.592975] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 03:59:41 (1790755181) [14872.603735] do_IRQ: 2 callbacks suppressed [14872.603742] do_IRQ: 0.41 No irq handler for vector [14872.615507] do_IRQ: 0.41 No irq handler for vector [14872.623377] do_IRQ: 0.41 No irq handler for vector [14872.627963] do_IRQ: 0.41 No irq handler for vector [14872.631733] do_IRQ: 0.41 No irq handler for vector [14872.637179] do_IRQ: 0.41 No irq handler for vector [14872.640294] do_IRQ: 0.41 No irq handler for vector [14872.645768] do_IRQ: 0.41 No irq handler for vector [14872.649635] do_IRQ: 0.41 No irq handler for vector [14872.653964] do_IRQ: 0.41 No irq handler for vector [14874.089898] LustreError: 121879:0:(fld_handler.c:268:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [14874.099964] LustreError: 121879:0:(fld_handler.c:268:fld_server_lookup()) Skipped 3 previous similar messages [14882.312928] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 03:59:51 (1790755191) [14882.331760] do_IRQ: 3 callbacks suppressed [14882.331767] do_IRQ: 0.41 No irq handler for vector [14882.338699] do_IRQ: 0.41 No irq handler for vector [14882.341699] do_IRQ: 0.41 No irq handler for vector [14882.348750] do_IRQ: 0.41 No irq handler for vector [14882.362502] do_IRQ: 0.41 No irq handler for vector [14882.368302] do_IRQ: 0.41 No irq handler for vector [14882.371613] do_IRQ: 0.41 No irq handler for vector [14882.374652] do_IRQ: 0.41 No irq handler for vector [14882.377888] do_IRQ: 0.41 No irq handler for vector [14882.381597] do_IRQ: 0.41 No irq handler for vector [14888.793331] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 03:59:57 (1790755197) [14888.799864] do_IRQ: 1 callbacks suppressed [14888.799873] do_IRQ: 0.41 No irq handler for vector [14888.811259] do_IRQ: 0.41 No irq handler for vector [14888.814642] do_IRQ: 0.41 No irq handler for vector [14888.816979] do_IRQ: 0.41 No irq handler for vector [14888.820796] do_IRQ: 0.41 No irq handler for vector [14888.827263] do_IRQ: 0.41 No irq handler for vector [14888.832341] do_IRQ: 0.41 No irq handler for vector [14888.835684] do_IRQ: 0.41 No irq handler for vector [14888.839814] do_IRQ: 0.41 No irq handler for vector [14888.844928] do_IRQ: 0.41 No irq handler for vector [14897.754793] Lustre: DEBUG MARKER: == sanity test 154db: fid is stored in dir entries ======= 04:00:06 (1790755206) [14897.772250] do_IRQ: 1 callbacks suppressed [14897.772261] do_IRQ: 0.41 No irq handler for vector [14897.790924] do_IRQ: 0.41 No irq handler for vector [14897.799942] do_IRQ: 0.41 No irq handler for vector [14897.807877] do_IRQ: 0.41 No irq handler for vector [14897.815637] do_IRQ: 0.41 No irq handler for vector [14897.825568] do_IRQ: 0.41 No irq handler for vector [14897.832020] do_IRQ: 0.41 No irq handler for vector [14897.838558] do_IRQ: 0.41 No irq handler for vector [14897.841738] do_IRQ: 0.41 No irq handler for vector [14897.847401] do_IRQ: 0.41 No irq handler for vector [14909.700880] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 04:00:18 (1790755218) [14909.712414] do_IRQ: 1 callbacks suppressed [14909.712421] do_IRQ: 0.41 No irq handler for vector [14909.723826] do_IRQ: 0.41 No irq handler for vector [14909.732329] do_IRQ: 0.41 No irq handler for vector [14909.738259] do_IRQ: 0.41 No irq handler for vector [14909.744150] do_IRQ: 0.41 No irq handler for vector [14909.748315] do_IRQ: 0.41 No irq handler for vector [14909.757308] do_IRQ: 0.41 No irq handler for vector [14909.764570] do_IRQ: 0.41 No irq handler for vector [14909.773280] do_IRQ: 0.41 No irq handler for vector [14909.785858] do_IRQ: 0.41 No irq handler for vector [14917.050269] Lustre: DEBUG MARKER: == sanity test 154ea: .lustre is not returned by readdir (2) ========================================================== 04:00:25 (1790755225) [14917.057233] do_IRQ: 1 callbacks suppressed [14917.057244] do_IRQ: 0.41 No irq handler for vector [14917.073605] do_IRQ: 0.41 No irq handler for vector [14917.075463] do_IRQ: 0.41 No irq handler for vector [14917.077646] do_IRQ: 0.41 No irq handler for vector [14917.079831] do_IRQ: 0.41 No irq handler for vector [14917.082249] do_IRQ: 0.41 No irq handler for vector [14917.084776] do_IRQ: 0.41 No irq handler for vector [14917.087283] do_IRQ: 0.41 No irq handler for vector [14917.090361] do_IRQ: 0.41 No irq handler for vector [14917.092784] do_IRQ: 0.41 No irq handler for vector [14971.834954] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 04:01:20 (1790755280) [14971.839926] do_IRQ: 1 callbacks suppressed [14971.839937] do_IRQ: 0.41 No irq handler for vector [14971.841969] do_IRQ: 0.41 No irq handler for vector [14971.848523] do_IRQ: 0.41 No irq handler for vector [14971.850813] do_IRQ: 0.41 No irq handler for vector [14971.853148] do_IRQ: 0.41 No irq handler for vector [14971.855491] do_IRQ: 0.41 No irq handler for vector [14971.857527] do_IRQ: 0.41 No irq handler for vector [14971.861055] do_IRQ: 0.41 No irq handler for vector [14971.865114] do_IRQ: 0.41 No irq handler for vector [14971.869259] do_IRQ: 0.41 No irq handler for vector [14978.703569] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 04:01:28 (1790755288) [14978.711075] do_IRQ: 1 callbacks suppressed [14978.711086] do_IRQ: 0.41 No irq handler for vector [14978.720198] do_IRQ: 0.41 No irq handler for vector [14978.725067] do_IRQ: 0.41 No irq handler for vector [14978.730185] do_IRQ: 0.41 No irq handler for vector [14978.733267] do_IRQ: 0.41 No irq handler for vector [14978.736163] do_IRQ: 0.41 No irq handler for vector [14978.739852] do_IRQ: 0.41 No irq handler for vector [14978.744467] do_IRQ: 0.41 No irq handler for vector [14978.747350] do_IRQ: 0.41 No irq handler for vector [14978.753896] do_IRQ: 0.41 No irq handler for vector [16123.144521] Lustre: DEBUG MARKER: == sanity test 154h: Verify interactive path2fid ========= 04:20:32 (1790756432) [16123.148950] do_IRQ: 1 callbacks suppressed [16123.148960] do_IRQ: 0.41 No irq handler for vector [16123.163954] do_IRQ: 0.41 No irq handler for vector [16123.174090] do_IRQ: 0.41 No irq handler for vector [16123.180858] do_IRQ: 0.41 No irq handler for vector [16123.185510] do_IRQ: 0.41 No irq handler for vector [16123.188272] do_IRQ: 0.41 No irq handler for vector [16123.193595] do_IRQ: 0.41 No irq handler for vector [16123.197296] do_IRQ: 0.41 No irq handler for vector [16123.201279] do_IRQ: 0.41 No irq handler for vector [16123.205029] do_IRQ: 0.41 No irq handler for vector [16130.175549] Lustre: DEBUG MARKER: == sanity test 154i: fid2path for path longer than PATH_MAX ========================================================== 04:20:38 (1790756438) [16130.199445] do_IRQ: 1 callbacks suppressed [16130.199453] do_IRQ: 0.41 No irq handler for vector [16130.211766] do_IRQ: 0.41 No irq handler for vector [16130.217145] do_IRQ: 0.41 No irq handler for vector [16130.220595] do_IRQ: 0.41 No irq handler for vector [16130.223534] do_IRQ: 0.41 No irq handler for vector [16130.226567] do_IRQ: 0.41 No irq handler for vector [16130.234564] do_IRQ: 0.41 No irq handler for vector [16130.242123] do_IRQ: 0.41 No irq handler for vector [16130.243570] do_IRQ: 0.41 No irq handler for vector [16130.252031] do_IRQ: 0.41 No irq handler for vector [16170.240110] Lustre: DEBUG MARKER: == sanity test 154j: fid2path for long path crossing MDT boundary ========================================================== 04:21:19 (1790756479) [16170.259763] do_IRQ: 1 callbacks suppressed [16170.259773] do_IRQ: 0.41 No irq handler for vector [16170.261331] do_IRQ: 0.41 No irq handler for vector [16170.266296] do_IRQ: 0.41 No irq handler for vector [16170.268849] do_IRQ: 0.41 No irq handler for vector [16170.271867] do_IRQ: 0.41 No irq handler for vector [16170.277728] do_IRQ: 0.41 No irq handler for vector [16170.283825] do_IRQ: 0.41 No irq handler for vector [16170.287313] do_IRQ: 0.41 No irq handler for vector [16170.290225] do_IRQ: 0.41 No irq handler for vector [16170.292982] do_IRQ: 0.41 No irq handler for vector [16192.358612] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 04:21:41 (1790756501) [16192.364757] do_IRQ: 2 callbacks suppressed [16192.364767] do_IRQ: 0.41 No irq handler for vector [16192.374436] do_IRQ: 0.41 No irq handler for vector [16192.376224] do_IRQ: 0.41 No irq handler for vector [16192.378266] do_IRQ: 0.41 No irq handler for vector [16192.381929] do_IRQ: 0.41 No irq handler for vector [16192.384644] do_IRQ: 0.41 No irq handler for vector [16192.387214] do_IRQ: 0.41 No irq handler for vector [16192.390165] do_IRQ: 0.41 No irq handler for vector [16192.392483] do_IRQ: 0.41 No irq handler for vector [16192.395248] do_IRQ: 0.41 No irq handler for vector [16205.662311] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 04:21:54 (1790756514) [16205.671565] do_IRQ: 1 callbacks suppressed [16205.671577] do_IRQ: 0.41 No irq handler for vector [16205.681354] do_IRQ: 0.41 No irq handler for vector [16205.683939] do_IRQ: 0.41 No irq handler for vector [16205.686478] do_IRQ: 0.41 No irq handler for vector [16205.693556] do_IRQ: 0.41 No irq handler for vector [16205.695927] do_IRQ: 0.41 No irq handler for vector [16205.699430] do_IRQ: 0.41 No irq handler for vector [16205.702842] do_IRQ: 0.41 No irq handler for vector [16205.705816] do_IRQ: 0.41 No irq handler for vector [16205.708615] do_IRQ: 0.41 No irq handler for vector [16218.025821] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 04:22:07 (1790756527) [16218.033879] do_IRQ: 1 callbacks suppressed [16218.033890] do_IRQ: 0.41 No irq handler for vector [16218.054935] do_IRQ: 0.41 No irq handler for vector [16218.064594] do_IRQ: 0.41 No irq handler for vector [16218.065942] do_IRQ: 0.41 No irq handler for vector [16218.067322] do_IRQ: 0.41 No irq handler for vector [16218.075975] do_IRQ: 0.41 No irq handler for vector [16218.080191] do_IRQ: 0.41 No irq handler for vector [16218.085417] do_IRQ: 0.41 No irq handler for vector [16218.089123] do_IRQ: 0.41 No irq handler for vector [16218.095756] do_IRQ: 0.41 No irq handler for vector [16231.436569] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 04:22:20 (1790756540) [16231.448064] do_IRQ: 1 callbacks suppressed [16231.448072] do_IRQ: 0.41 No irq handler for vector [16231.460750] do_IRQ: 0.41 No irq handler for vector [16231.465924] do_IRQ: 0.41 No irq handler for vector [16231.471510] do_IRQ: 0.41 No irq handler for vector [16231.473379] do_IRQ: 0.41 No irq handler for vector [16231.478095] do_IRQ: 0.41 No irq handler for vector [16231.483279] do_IRQ: 0.41 No irq handler for vector [16231.486440] do_IRQ: 0.41 No irq handler for vector [16231.491336] do_IRQ: 0.41 No irq handler for vector [16231.500059] do_IRQ: 0.41 No irq handler for vector [16243.960405] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 04:22:33 (1790756553) [16243.983433] do_IRQ: 1 callbacks suppressed [16243.983443] do_IRQ: 0.41 No irq handler for vector [16243.991792] do_IRQ: 0.41 No irq handler for vector [16244.003522] do_IRQ: 0.41 No irq handler for vector [16244.011763] do_IRQ: 0.41 No irq handler for vector [16244.017745] do_IRQ: 0.41 No irq handler for vector [16244.023293] do_IRQ: 0.41 No irq handler for vector [16244.027900] do_IRQ: 0.41 No irq handler for vector [16244.032633] do_IRQ: 0.41 No irq handler for vector [16244.038596] do_IRQ: 0.41 No irq handler for vector [16244.043900] do_IRQ: 0.41 No irq handler for vector [16292.738550] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 04:23:21 (1790756601) [16292.752153] do_IRQ: 2 callbacks suppressed [16292.752162] do_IRQ: 0.41 No irq handler for vector [16292.756662] do_IRQ: 0.41 No irq handler for vector [16292.765659] do_IRQ: 0.41 No irq handler for vector [16292.770770] do_IRQ: 0.41 No irq handler for vector [16292.779124] do_IRQ: 0.41 No irq handler for vector [16292.786037] do_IRQ: 0.41 No irq handler for vector [16292.789279] do_IRQ: 0.41 No irq handler for vector [16292.794272] do_IRQ: 0.41 No irq handler for vector [16292.797483] do_IRQ: 0.41 No irq handler for vector [16292.801085] do_IRQ: 0.41 No irq handler for vector [16335.867300] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 04:24:04 (1790756644) [16335.891021] do_IRQ: 2 callbacks suppressed [16335.891030] do_IRQ: 0.41 No irq handler for vector [16335.903665] do_IRQ: 0.41 No irq handler for vector [16335.914821] do_IRQ: 0.41 No irq handler for vector [16335.918761] do_IRQ: 0.41 No irq handler for vector [16335.927580] do_IRQ: 0.41 No irq handler for vector [16335.930489] do_IRQ: 0.41 No irq handler for vector [16335.933988] do_IRQ: 0.41 No irq handler for vector [16335.940095] do_IRQ: 0.41 No irq handler for vector [16335.944940] do_IRQ: 0.41 No irq handler for vector [16335.949092] do_IRQ: 0.41 No irq handler for vector [16379.412207] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 04:24:48 (1790756688) [16379.423299] do_IRQ: 1 callbacks suppressed [16379.423310] do_IRQ: 0.41 No irq handler for vector [16379.431817] do_IRQ: 0.41 No irq handler for vector [16379.445101] do_IRQ: 0.41 No irq handler for vector [16379.447868] do_IRQ: 0.41 No irq handler for vector [16379.450389] do_IRQ: 0.41 No irq handler for vector [16379.453279] do_IRQ: 0.41 No irq handler for vector [16379.455890] do_IRQ: 0.41 No irq handler for vector [16379.458112] do_IRQ: 0.41 No irq handler for vector [16379.461272] do_IRQ: 0.41 No irq handler for vector [16379.463421] do_IRQ: 0.41 No irq handler for vector [16425.735977] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 04:25:34 (1790756734) [16425.744891] do_IRQ: 1 callbacks suppressed [16425.744915] do_IRQ: 0.41 No irq handler for vector [16425.757777] do_IRQ: 0.41 No irq handler for vector [16425.765915] do_IRQ: 0.41 No irq handler for vector [16425.772250] do_IRQ: 0.41 No irq handler for vector [16425.776420] do_IRQ: 0.41 No irq handler for vector [16425.780619] do_IRQ: 0.41 No irq handler for vector [16425.789951] do_IRQ: 0.41 No irq handler for vector [16425.792791] do_IRQ: 0.41 No irq handler for vector [16425.799783] do_IRQ: 0.41 No irq handler for vector [16425.802961] do_IRQ: 0.41 No irq handler for vector [16435.840575] Lustre: DEBUG MARKER: Turn on read and write cache [16435.843862] do_IRQ: 1 callbacks suppressed [16435.843872] do_IRQ: 0.41 No irq handler for vector [16435.854173] do_IRQ: 0.41 No irq handler for vector [16435.859892] do_IRQ: 0.41 No irq handler for vector [16435.862320] do_IRQ: 0.41 No irq handler for vector [16435.865741] do_IRQ: 0.41 No irq handler for vector [16435.872903] do_IRQ: 0.41 No irq handler for vector [16435.875753] do_IRQ: 0.41 No irq handler for vector [16435.878721] do_IRQ: 0.41 No irq handler for vector [16435.883582] do_IRQ: 0.41 No irq handler for vector [16435.890862] do_IRQ: 0.41 No irq handler for vector [16439.711776] Lustre: DEBUG MARKER: Write data and read it back. [16441.129835] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [16441.137889] do_IRQ: 2 callbacks suppressed [16441.137898] do_IRQ: 0.41 No irq handler for vector [16441.145551] do_IRQ: 0.41 No irq handler for vector [16441.154869] do_IRQ: 0.41 No irq handler for vector [16441.158662] do_IRQ: 0.41 No irq handler for vector [16441.161166] do_IRQ: 0.41 No irq handler for vector [16441.164484] do_IRQ: 0.41 No irq handler for vector [16441.170233] do_IRQ: 0.41 No irq handler for vector [16441.171980] do_IRQ: 0.41 No irq handler for vector [16441.181900] do_IRQ: 0.41 No irq handler for vector [16441.185342] do_IRQ: 0.41 No irq handler for vector [16445.205433] Lustre: DEBUG MARKER: cache hits: before: 28721, after: 28724 [16447.001249] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [16447.010287] do_IRQ: 2 callbacks suppressed [16447.010299] do_IRQ: 0.41 No irq handler for vector [16447.021884] do_IRQ: 0.41 No irq handler for vector [16447.028449] do_IRQ: 0.41 No irq handler for vector [16447.036601] do_IRQ: 0.41 No irq handler for vector [16447.038941] do_IRQ: 0.41 No irq handler for vector [16447.052359] do_IRQ: 0.41 No irq handler for vector [16447.057511] do_IRQ: 0.41 No irq handler for vector [16447.060002] do_IRQ: 0.41 No irq handler for vector [16447.064882] do_IRQ: 0.41 No irq handler for vector [16447.069291] do_IRQ: 0.41 No irq handler for vector [16449.969489] Lustre: DEBUG MARKER: cache hits:: before: 28724, after: 28727 [16451.656074] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [16455.731354] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [16455.735363] do_IRQ: 3 callbacks suppressed [16455.735376] do_IRQ: 0.41 No irq handler for vector [16455.741832] do_IRQ: 0.41 No irq handler for vector [16455.745440] do_IRQ: 0.41 No irq handler for vector [16455.750394] do_IRQ: 0.41 No irq handler for vector [16455.754915] do_IRQ: 0.41 No irq handler for vector [16455.757168] do_IRQ: 0.41 No irq handler for vector [16455.764501] do_IRQ: 0.41 No irq handler for vector [16455.777939] do_IRQ: 0.41 No irq handler for vector [16455.779448] do_IRQ: 0.41 No irq handler for vector [16455.782839] do_IRQ: 0.41 No irq handler for vector [16459.455453] Lustre: DEBUG MARKER: cache hits:: before: 28727, after: 28730 [16461.206916] Lustre: DEBUG MARKER: Write data and read it back. [16461.213574] do_IRQ: 2 callbacks suppressed [16461.213585] do_IRQ: 0.41 No irq handler for vector [16461.225526] do_IRQ: 0.41 No irq handler for vector [16461.228900] do_IRQ: 0.41 No irq handler for vector [16461.233134] do_IRQ: 0.41 No irq handler for vector [16461.237155] do_IRQ: 0.41 No irq handler for vector [16461.240086] do_IRQ: 0.41 No irq handler for vector [16461.243119] do_IRQ: 0.41 No irq handler for vector [16461.246272] do_IRQ: 0.41 No irq handler for vector [16461.248934] do_IRQ: 0.41 No irq handler for vector [16461.251864] do_IRQ: 0.41 No irq handler for vector [16462.643642] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [16466.183361] Lustre: DEBUG MARKER: cache hits:: before: 28730, after: 28733 [16467.734697] Lustre: DEBUG MARKER: Turn off read and write cache [16467.739117] do_IRQ: 3 callbacks suppressed [16467.739127] do_IRQ: 0.41 No irq handler for vector [16467.743582] do_IRQ: 0.41 No irq handler for vector [16467.749823] do_IRQ: 0.41 No irq handler for vector [16467.752883] do_IRQ: 0.41 No irq handler for vector [16467.755672] do_IRQ: 0.41 No irq handler for vector [16467.758745] do_IRQ: 0.41 No irq handler for vector [16467.761320] do_IRQ: 0.41 No irq handler for vector [16467.763590] do_IRQ: 0.41 No irq handler for vector [16467.769449] do_IRQ: 0.41 No irq handler for vector [16467.772471] do_IRQ: 0.41 No irq handler for vector [16471.374470] Lustre: DEBUG MARKER: Write data and read it back [16472.725703] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [16475.876542] Lustre: DEBUG MARKER: cache hits:: before: 28733, after: 28733 [16475.880133] do_IRQ: 4 callbacks suppressed [16475.880144] do_IRQ: 0.41 No irq handler for vector [16475.890693] do_IRQ: 0.41 No irq handler for vector [16475.893904] do_IRQ: 0.41 No irq handler for vector [16475.896973] do_IRQ: 0.41 No irq handler for vector [16475.901691] do_IRQ: 0.41 No irq handler for vector [16475.907014] do_IRQ: 0.41 No irq handler for vector [16475.912556] do_IRQ: 0.41 No irq handler for vector [16475.917201] do_IRQ: 0.41 No irq handler for vector [16475.920805] do_IRQ: 0.41 No irq handler for vector [16475.926190] do_IRQ: 0.41 No irq handler for vector [16477.173416] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [16480.337057] Lustre: DEBUG MARKER: Write data and read it back [16481.730743] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [16481.734625] do_IRQ: 3 callbacks suppressed [16481.734635] do_IRQ: 0.41 No irq handler for vector [16481.744359] do_IRQ: 0.41 No irq handler for vector [16481.746709] do_IRQ: 0.41 No irq handler for vector [16481.754033] do_IRQ: 0.41 No irq handler for vector [16481.758561] do_IRQ: 0.41 No irq handler for vector [16481.762248] do_IRQ: 0.41 No irq handler for vector [16481.770493] do_IRQ: 0.41 No irq handler for vector [16481.772939] do_IRQ: 0.41 No irq handler for vector [16481.777087] do_IRQ: 0.41 No irq handler for vector [16481.781832] do_IRQ: 0.41 No irq handler for vector [16485.385968] Lustre: DEBUG MARKER: cache hits:: before: 28733, after: 28733 [16486.982784] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [16486.988215] do_IRQ: 2 callbacks suppressed [16486.988222] do_IRQ: 0.41 No irq handler for vector [16486.998796] do_IRQ: 0.41 No irq handler for vector [16487.000617] do_IRQ: 0.41 No irq handler for vector [16487.007479] do_IRQ: 0.41 No irq handler for vector [16487.014685] do_IRQ: 0.41 No irq handler for vector [16487.018943] do_IRQ: 0.41 No irq handler for vector [16487.023908] do_IRQ: 0.41 No irq handler for vector [16487.027827] do_IRQ: 0.41 No irq handler for vector [16487.037433] do_IRQ: 0.41 No irq handler for vector [16487.045596] do_IRQ: 0.41 No irq handler for vector [16490.324691] Lustre: DEBUG MARKER: cache hits:: before: 28733, after: 28736 [16498.804306] Lustre: DEBUG MARKER: == sanity test 157a: llapi pool pinning API tests ======== 04:26:47 (1790756807) [16498.817001] do_IRQ: 2 callbacks suppressed [16498.817009] do_IRQ: 0.41 No irq handler for vector [16498.823137] do_IRQ: 0.41 No irq handler for vector [16498.835249] do_IRQ: 0.41 No irq handler for vector [16498.841762] do_IRQ: 0.41 No irq handler for vector [16498.847551] do_IRQ: 0.41 No irq handler for vector [16498.851653] do_IRQ: 0.41 No irq handler for vector [16498.864436] do_IRQ: 0.41 No irq handler for vector [16498.871635] do_IRQ: 0.41 No irq handler for vector [16498.873251] do_IRQ: 0.41 No irq handler for vector [16498.884104] do_IRQ: 0.41 No irq handler for vector [16507.311678] Lustre: DEBUG MARKER: == sanity test 157b: lustre.pin inheritance on create ==== 04:26:56 (1790756816) [16507.318768] do_IRQ: 1 callbacks suppressed [16507.318777] do_IRQ: 0.41 No irq handler for vector [16507.321588] do_IRQ: 0.41 No irq handler for vector [16507.327018] do_IRQ: 0.41 No irq handler for vector [16507.330513] do_IRQ: 0.41 No irq handler for vector [16507.333650] do_IRQ: 0.41 No irq handler for vector [16507.336140] do_IRQ: 0.41 No irq handler for vector [16507.338569] do_IRQ: 0.41 No irq handler for vector [16507.342009] do_IRQ: 0.41 No irq handler for vector [16507.344862] do_IRQ: 0.41 No irq handler for vector [16507.347659] do_IRQ: 0.41 No irq handler for vector [16517.344222] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 04:27:06 (1790756826) [16517.354316] do_IRQ: 1 callbacks suppressed [16517.354327] do_IRQ: 0.41 No irq handler for vector [16517.376109] do_IRQ: 0.41 No irq handler for vector [16517.377883] do_IRQ: 0.41 No irq handler for vector [16517.393305] do_IRQ: 0.41 No irq handler for vector [16517.412604] do_IRQ: 0.41 No irq handler for vector [16517.426074] do_IRQ: 0.41 No irq handler for vector [16517.430956] do_IRQ: 0.41 No irq handler for vector [16517.437550] do_IRQ: 0.41 No irq handler for vector [16517.444829] do_IRQ: 0.41 No irq handler for vector [16517.447776] do_IRQ: 0.41 No irq handler for vector [16520.170067] Lustre: lustre-MDD0000: changelog on [16522.993738] Lustre: lustre-MDD0001: changelog on [16522.996844] do_IRQ: 2 callbacks suppressed [16522.996852] do_IRQ: 0.41 No irq handler for vector [16523.004970] do_IRQ: 0.41 No irq handler for vector [16523.009219] do_IRQ: 0.41 No irq handler for vector [16523.014267] do_IRQ: 0.41 No irq handler for vector [16523.017179] do_IRQ: 0.41 No irq handler for vector [16523.021795] do_IRQ: 0.41 No irq handler for vector [16523.028177] do_IRQ: 0.41 No irq handler for vector [16523.033867] do_IRQ: 0.41 No irq handler for vector [16523.035467] do_IRQ: 0.41 No irq handler for vector [16523.042255] do_IRQ: 0.41 No irq handler for vector [16538.958534] Lustre: Failing over lustre-MDT0000 [16538.963064] do_IRQ: 1 callbacks suppressed [16538.963070] do_IRQ: 0.41 No irq handler for vector [16538.975761] do_IRQ: 0.41 No irq handler for vector [16538.977427] do_IRQ: 0.41 No irq handler for vector [16538.980858] do_IRQ: 0.41 No irq handler for vector [16538.984096] do_IRQ: 0.41 No irq handler for vector [16538.986501] do_IRQ: 0.41 No irq handler for vector [16538.988974] do_IRQ: 0.41 No irq handler for vector [16538.992009] do_IRQ: 0.41 No irq handler for vector [16538.994235] do_IRQ: 0.41 No irq handler for vector [16538.996447] do_IRQ: 0.41 No irq handler for vector [16539.450934] Lustre: server umount lustre-MDT0000 complete [16541.393234] LustreError: 121868:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [16541.666261] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16541.699208] Lustre: Skipped 2 previous similar messages [16546.497104] LustreError: 121869:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [16546.539230] LustreError: 121869:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [16546.539609] do_IRQ: 5 callbacks suppressed [16546.539618] do_IRQ: 0.41 No irq handler for vector [16546.557225] do_IRQ: 0.41 No irq handler for vector [16546.561138] do_IRQ: 0.41 No irq handler for vector [16546.564609] do_IRQ: 0.41 No irq handler for vector [16546.571619] do_IRQ: 0.41 No irq handler for vector [16546.575190] do_IRQ: 0.41 No irq handler for vector [16546.579583] do_IRQ: 0.41 No irq handler for vector [16546.585038] do_IRQ: 0.41 No irq handler for vector [16546.591078] do_IRQ: 0.41 No irq handler for vector [16546.594198] do_IRQ: 0.41 No irq handler for vector [16548.283400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16548.480163] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16548.786222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16548.831590] Lustre: lustre-MDD0000: changelog on [16548.864471] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [16551.619903] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [16551.633261] do_IRQ: 6 callbacks suppressed [16551.633325] do_IRQ: 0.41 No irq handler for vector [16551.650070] do_IRQ: 0.41 No irq handler for vector [16551.659859] do_IRQ: 0.41 No irq handler for vector [16551.663325] do_IRQ: 0.41 No irq handler for vector [16551.665938] do_IRQ: 0.41 No irq handler for vector [16551.673402] do_IRQ: 0.41 No irq handler for vector [16551.682416] do_IRQ: 0.41 No irq handler for vector [16551.693570] do_IRQ: 0.41 No irq handler for vector [16551.699977] do_IRQ: 0.41 No irq handler for vector [16551.704650] do_IRQ: 0.41 No irq handler for vector [16553.857754] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [16553.968016] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [16554.614425] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [16554.618776] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [16554.629428] Lustre: Skipped 2 previous similar messages [16554.668492] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:24093 to 0x2c0000403:24129) [16554.670080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:30433 to 0x280000403:30465) [16558.570813] Lustre: lustre-MDD0000: changelog off [16558.574753] do_IRQ: 8 callbacks suppressed [16558.574762] do_IRQ: 0.41 No irq handler for vector [16558.587031] do_IRQ: 0.41 No irq handler for vector [16558.594552] do_IRQ: 0.41 No irq handler for vector [16558.596428] do_IRQ: 0.41 No irq handler for vector [16558.605312] do_IRQ: 0.41 No irq handler for vector [16558.609310] do_IRQ: 0.41 No irq handler for vector [16558.611193] do_IRQ: 0.41 No irq handler for vector [16558.618117] do_IRQ: 0.41 No irq handler for vector [16558.621643] do_IRQ: 0.41 No irq handler for vector [16558.623946] do_IRQ: 0.41 No irq handler for vector [16560.364581] Lustre: lustre-MDD0001: changelog off [16574.901834] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 04:28:03 (1790756883) [16574.918032] do_IRQ: 2 callbacks suppressed [16574.918043] do_IRQ: 0.41 No irq handler for vector [16574.943201] do_IRQ: 0.41 No irq handler for vector [16574.945025] do_IRQ: 0.41 No irq handler for vector [16574.954036] do_IRQ: 0.41 No irq handler for vector [16574.964539] do_IRQ: 0.41 No irq handler for vector [16574.967897] do_IRQ: 0.41 No irq handler for vector [16574.975118] do_IRQ: 0.41 No irq handler for vector [16574.985116] do_IRQ: 0.41 No irq handler for vector [16574.995584] do_IRQ: 0.41 No irq handler for vector [16575.000724] do_IRQ: 0.41 No irq handler for vector [16578.058313] Lustre: lustre-MDD0000: changelog on [16590.834742] Lustre: lustre-MDD0001: changelog off [16590.837435] do_IRQ: 2 callbacks suppressed [16590.837445] do_IRQ: 0.41 No irq handler for vector [16590.849128] do_IRQ: 0.41 No irq handler for vector [16590.854572] do_IRQ: 0.41 No irq handler for vector [16590.860531] do_IRQ: 0.41 No irq handler for vector [16590.865268] do_IRQ: 0.41 No irq handler for vector [16590.868797] do_IRQ: 0.41 No irq handler for vector [16590.872937] do_IRQ: 0.41 No irq handler for vector [16590.875795] do_IRQ: 0.41 No irq handler for vector [16590.878561] do_IRQ: 0.41 No irq handler for vector [16590.882054] do_IRQ: 0.41 No irq handler for vector [16593.576631] Lustre: lustre-MDD0000: changelog off [16596.069179] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 04:28:25 (1790756905) [16596.083369] do_IRQ: 2 callbacks suppressed [16596.083376] do_IRQ: 0.41 No irq handler for vector [16596.088788] do_IRQ: 0.41 No irq handler for vector [16596.098630] do_IRQ: 0.41 No irq handler for vector [16596.105390] do_IRQ: 0.41 No irq handler for vector [16596.109700] do_IRQ: 0.41 No irq handler for vector [16596.112810] do_IRQ: 0.41 No irq handler for vector [16596.114911] do_IRQ: 0.41 No irq handler for vector [16596.117450] do_IRQ: 0.41 No irq handler for vector [16596.119883] do_IRQ: 0.41 No irq handler for vector [16596.122713] do_IRQ: 0.41 No irq handler for vector [16598.872530] Lustre: lustre-MDD0000: changelog on [16598.874814] Lustre: Skipped 1 previous similar message [16612.440770] Lustre: lustre-MDD0001: changelog off [16612.448163] do_IRQ: 3 callbacks suppressed [16612.448228] do_IRQ: 0.41 No irq handler for vector [16612.450554] do_IRQ: 0.41 No irq handler for vector [16612.452956] do_IRQ: 0.41 No irq handler for vector [16612.458099] do_IRQ: 0.41 No irq handler for vector [16612.461638] do_IRQ: 0.41 No irq handler for vector [16612.465621] do_IRQ: 0.41 No irq handler for vector [16612.468669] do_IRQ: 0.41 No irq handler for vector [16612.471644] do_IRQ: 0.41 No irq handler for vector [16612.474591] do_IRQ: 0.41 No irq handler for vector [16612.477431] do_IRQ: 0.41 No irq handler for vector [16617.492181] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 04:28:46 (1790756926) [16617.508678] do_IRQ: 2 callbacks suppressed [16617.508688] do_IRQ: 0.41 No irq handler for vector [16617.526228] do_IRQ: 0.41 No irq handler for vector [16617.529432] do_IRQ: 0.41 No irq handler for vector [16617.537588] do_IRQ: 0.41 No irq handler for vector [16617.545737] do_IRQ: 0.41 No irq handler for vector [16617.549531] do_IRQ: 0.41 No irq handler for vector [16617.555245] do_IRQ: 0.41 No irq handler for vector [16617.563461] do_IRQ: 0.41 No irq handler for vector [16617.571863] do_IRQ: 0.41 No irq handler for vector [16617.574382] do_IRQ: 0.41 No irq handler for vector [16620.946395] Lustre: lustre-MDD0000: changelog on [16620.948921] Lustre: Skipped 1 previous similar message [16632.203235] Lustre: lustre-MDD0001: changelog off [16632.206489] do_IRQ: 3 callbacks suppressed [16632.206498] do_IRQ: 0.41 No irq handler for vector [16632.209739] do_IRQ: 0.41 No irq handler for vector [16632.219708] do_IRQ: 0.41 No irq handler for vector [16632.225242] do_IRQ: 0.41 No irq handler for vector [16632.229592] do_IRQ: 0.41 No irq handler for vector [16632.233646] do_IRQ: 0.41 No irq handler for vector [16632.238503] do_IRQ: 0.41 No irq handler for vector [16632.242258] do_IRQ: 0.41 No irq handler for vector [16632.245881] do_IRQ: 0.41 No irq handler for vector [16632.250498] do_IRQ: 0.41 No irq handler for vector [16632.256237] Lustre: Skipped 1 previous similar message [16637.632530] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 04:29:06 (1790756946) [16637.646226] do_IRQ: 3 callbacks suppressed [16637.646236] do_IRQ: 0.41 No irq handler for vector [16637.660275] do_IRQ: 0.41 No irq handler for vector [16637.663320] do_IRQ: 0.41 No irq handler for vector [16637.666657] do_IRQ: 0.41 No irq handler for vector [16637.671721] do_IRQ: 0.41 No irq handler for vector [16637.677364] do_IRQ: 0.41 No irq handler for vector [16637.680559] do_IRQ: 0.41 No irq handler for vector [16637.684417] do_IRQ: 0.41 No irq handler for vector [16637.687354] do_IRQ: 0.41 No irq handler for vector [16637.691606] do_IRQ: 0.41 No irq handler for vector [16640.838666] Lustre: lustre-MDD0000: changelog on [16640.847922] Lustre: Skipped 1 previous similar message [16652.388706] Lustre: lustre-MDD0001: changelog off [16652.395475] do_IRQ: 3 callbacks suppressed [16652.395486] do_IRQ: 0.41 No irq handler for vector [16652.399547] do_IRQ: 0.41 No irq handler for vector [16652.407597] do_IRQ: 0.41 No irq handler for vector [16652.411457] do_IRQ: 0.41 No irq handler for vector [16652.414831] do_IRQ: 0.41 No irq handler for vector [16652.418814] do_IRQ: 0.41 No irq handler for vector [16652.422237] do_IRQ: 0.41 No irq handler for vector [16652.425146] do_IRQ: 0.41 No irq handler for vector [16652.428283] do_IRQ: 0.41 No irq handler for vector [16652.431111] do_IRQ: 0.41 No irq handler for vector [16652.436321] Lustre: Skipped 1 previous similar message [16657.065810] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 04:29:26 (1790756966) [16669.778988] Lustre: DEBUG MARKER: 1790756978: creating first dirs [16669.782791] do_IRQ: 4 callbacks suppressed [16669.782800] do_IRQ: 0.41 No irq handler for vector [16669.787598] do_IRQ: 0.41 No irq handler for vector [16669.794392] do_IRQ: 0.41 No irq handler for vector [16669.797120] do_IRQ: 0.41 No irq handler for vector [16669.801818] do_IRQ: 0.41 No irq handler for vector [16669.805139] do_IRQ: 0.41 No irq handler for vector [16669.809851] do_IRQ: 0.41 No irq handler for vector [16669.813093] do_IRQ: 0.41 No irq handler for vector [16669.816482] do_IRQ: 0.41 No irq handler for vector [16669.819938] do_IRQ: 0.41 No irq handler for vector [16692.885562] Lustre: *** cfs_fail_loc=1313, val=3*** [16692.888575] Lustre: 121869:0:(mdd_dir.c:995:mdd_changelog_emrg_cleanup()) lustre-MDD0000: changelog has only 3 free catalog entries [16692.888837] do_IRQ: 2 callbacks suppressed [16692.888841] do_IRQ: 0.41 No irq handler for vector [16692.896588] Lustre: 121869:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [16692.907299] do_IRQ: 0.41 No irq handler for vector [16692.909731] do_IRQ: 0.41 No irq handler for vector [16692.912538] do_IRQ: 0.41 No irq handler for vector [16692.915178] do_IRQ: 0.41 No irq handler for vector [16692.917605] do_IRQ: 0.41 No irq handler for vector [16692.920500] do_IRQ: 0.41 No irq handler for vector [16692.923193] do_IRQ: 0.41 No irq handler for vector [16692.926330] do_IRQ: 0.41 No irq handler for vector [16692.929368] do_IRQ: 0.41 No irq handler for vector [16692.937340] Lustre: 221054:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl7 idle for 27s with 4 unprocessed records [16718.211061] Lustre: lustre-MDD0001: changelog off [16718.214350] do_IRQ: 2 callbacks suppressed [16718.214362] do_IRQ: 0.41 No irq handler for vector [16718.215570] Lustre: Skipped 1 previous similar message [16718.234434] do_IRQ: 0.41 No irq handler for vector [16718.238326] do_IRQ: 0.41 No irq handler for vector [16718.242309] do_IRQ: 0.41 No irq handler for vector [16718.247691] do_IRQ: 0.41 No irq handler for vector [16718.251467] do_IRQ: 0.41 No irq handler for vector [16718.254226] do_IRQ: 0.41 No irq handler for vector [16718.257475] do_IRQ: 0.41 No irq handler for vector [16718.264233] do_IRQ: 0.41 No irq handler for vector [16718.279414] do_IRQ: 0.41 No irq handler for vector [16723.463503] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 04:30:32 (1790757032) [16723.472540] do_IRQ: 1 callbacks suppressed [16723.472551] do_IRQ: 0.41 No irq handler for vector [16723.484884] do_IRQ: 0.41 No irq handler for vector [16723.487646] do_IRQ: 0.41 No irq handler for vector [16723.492292] do_IRQ: 0.41 No irq handler for vector [16723.495583] do_IRQ: 0.41 No irq handler for vector [16723.502408] do_IRQ: 0.41 No irq handler for vector [16723.505185] do_IRQ: 0.41 No irq handler for vector [16723.507664] do_IRQ: 0.41 No irq handler for vector [16723.510537] do_IRQ: 0.41 No irq handler for vector [16723.514157] do_IRQ: 0.41 No irq handler for vector [16726.706971] Lustre: lustre-MDD0000: changelog on [16726.713263] Lustre: Skipped 3 previous similar messages [16751.897855] Lustre: 121868:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [16751.923205] do_IRQ: 3 callbacks suppressed [16751.923213] do_IRQ: 0.41 No irq handler for vector [16751.928438] do_IRQ: 0.41 No irq handler for vector [16751.933370] Lustre: 121868:0:(mdd_dir.c:1082:mdd_changelog_store()) Skipped 1 previous similar message [16751.948806] do_IRQ: 0.41 No irq handler for vector [16751.954158] do_IRQ: 0.41 No irq handler for vector [16751.957839] do_IRQ: 0.41 No irq handler for vector [16751.966118] do_IRQ: 0.41 No irq handler for vector [16751.970573] do_IRQ: 0.41 No irq handler for vector [16751.973110] do_IRQ: 0.41 No irq handler for vector [16751.978068] do_IRQ: 0.41 No irq handler for vector [16751.979801] do_IRQ: 0.41 No irq handler for vector [16751.996915] Lustre: 223518:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl9 idle for 19s with 4 unprocessed records [16752.014563] Lustre: 223518:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [16777.826486] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 04:31:26 (1790757086) [16777.835471] do_IRQ: 3 callbacks suppressed [16777.835481] do_IRQ: 0.41 No irq handler for vector [16777.837621] do_IRQ: 0.41 No irq handler for vector [16777.840718] do_IRQ: 0.41 No irq handler for vector [16777.843964] do_IRQ: 0.41 No irq handler for vector [16777.846697] do_IRQ: 0.41 No irq handler for vector [16777.850452] do_IRQ: 0.41 No irq handler for vector [16777.853862] do_IRQ: 0.41 No irq handler for vector [16777.856820] do_IRQ: 0.41 No irq handler for vector [16777.859569] do_IRQ: 0.41 No irq handler for vector [16777.861775] do_IRQ: 0.41 No irq handler for vector [16813.765239] Lustre: *** cfs_fail_loc=1316, val=0*** [16813.767864] Lustre: 121870:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0001: simulate starting changelog garbage collection [16813.768214] do_IRQ: 2 callbacks suppressed [16813.768221] do_IRQ: 0.41 No irq handler for vector [16813.782462] Lustre: 121870:0:(mdd_dir.c:1082:mdd_changelog_store()) Skipped 1 previous similar message [16813.787444] Lustre: 225975:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0001: force deregister of changelog user cl11 idle for 26s with 3 unprocessed records [16813.788619] do_IRQ: 0.41 No irq handler for vector [16813.799705] do_IRQ: 0.41 No irq handler for vector [16813.802672] Lustre: 225975:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [16813.807374] do_IRQ: 0.41 No irq handler for vector [16813.809506] do_IRQ: 0.41 No irq handler for vector [16813.824658] do_IRQ: 0.41 No irq handler for vector [16813.832000] do_IRQ: 0.41 No irq handler for vector [16813.834665] do_IRQ: 0.41 No irq handler for vector [16813.839833] do_IRQ: 0.41 No irq handler for vector [16813.846766] do_IRQ: 0.41 No irq handler for vector [16815.663372] Lustre: Failing over lustre-MDT0001 [16815.921715] Lustre: Failing over lustre-MDT0000 [16817.878464] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.29@tcp (stopping) [16820.199171] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16820.215189] do_IRQ: 4 callbacks suppressed [16820.215198] do_IRQ: 0.41 No irq handler for vector [16820.219550] do_IRQ: 0.41 No irq handler for vector [16820.222947] do_IRQ: 0.41 No irq handler for vector [16820.225846] do_IRQ: 0.41 No irq handler for vector [16820.229929] do_IRQ: 0.41 No irq handler for vector [16820.233932] do_IRQ: 0.41 No irq handler for vector [16820.238678] do_IRQ: 0.41 No irq handler for vector [16820.241585] do_IRQ: 0.41 No irq handler for vector [16820.244635] Lustre: Skipped 3 previous similar messages [16820.249213] do_IRQ: 0.41 No irq handler for vector [16820.249494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16820.256003] do_IRQ: 0.41 No irq handler for vector [16820.256359] Lustre: Skipped 3 previous similar messages [16821.914245] Lustre: server umount lustre-MDT0001 complete [16822.008702] Lustre: server umount lustre-MDT0000 complete [16829.437708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16829.443357] do_IRQ: 3 callbacks suppressed [16829.443377] do_IRQ: 0.41 No irq handler for vector [16829.449446] do_IRQ: 0.41 No irq handler for vector [16829.451250] do_IRQ: 0.41 No irq handler for vector [16829.452839] do_IRQ: 0.41 No irq handler for vector [16829.455441] do_IRQ: 0.41 No irq handler for vector [16829.461136] do_IRQ: 0.41 No irq handler for vector [16829.463289] do_IRQ: 0.41 No irq handler for vector [16829.464954] do_IRQ: 0.41 No irq handler for vector [16829.469219] do_IRQ: 0.41 No irq handler for vector [16829.482587] do_IRQ: 0.41 No irq handler for vector [16829.573599] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16829.779894] LustreError: 123688:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16829.808711] LustreError: 123688:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [16829.928993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16829.966282] Lustre: lustre-MDD0000: changelog on [16829.969131] Lustre: Skipped 3 previous similar messages [16829.973548] Lustre: 227154:0:(mdd_device.c:628:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 33 to index 34, being cleared now [16829.990841] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [16833.940746] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [16836.067685] LustreError: 123689:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16836.083920] do_IRQ: 10 callbacks suppressed [16836.083931] do_IRQ: 0.41 No irq handler for vector [16836.087586] do_IRQ: 0.41 No irq handler for vector [16836.090639] do_IRQ: 0.41 No irq handler for vector [16836.100206] do_IRQ: 0.41 No irq handler for vector [16836.105233] do_IRQ: 0.41 No irq handler for vector [16836.110923] do_IRQ: 0.41 No irq handler for vector [16836.115958] do_IRQ: 0.41 No irq handler for vector [16836.121950] do_IRQ: 0.41 No irq handler for vector [16836.130712] do_IRQ: 0.41 No irq handler for vector [16836.132605] do_IRQ: 0.41 No irq handler for vector [16836.149659] LustreError: 123689:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16838.393640] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [16840.165913] LustreError: 227166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16840.171716] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16840.178033] LustreError: 227166:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [16841.537019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16841.548666] do_IRQ: 7 callbacks suppressed [16841.548675] do_IRQ: 0.41 No irq handler for vector [16841.551722] do_IRQ: 0.41 No irq handler for vector [16841.553697] do_IRQ: 0.41 No irq handler for vector [16841.555686] do_IRQ: 0.41 No irq handler for vector [16841.558202] do_IRQ: 0.41 No irq handler for vector [16841.560442] do_IRQ: 0.41 No irq handler for vector [16841.562440] do_IRQ: 0.41 No irq handler for vector [16841.564537] do_IRQ: 0.41 No irq handler for vector [16841.567224] do_IRQ: 0.41 No irq handler for vector [16841.569550] do_IRQ: 0.41 No irq handler for vector [16842.014649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16842.049391] Lustre: 227901:0:(mdd_device.c:628:mdd_changelog_llog_init()) lustre-MDD0001 : orphan changelog records found, starting from index 13 to index 14, being cleared now [16842.083200] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [16843.472112] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [16846.358685] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [16847.355345] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16847.368420] Lustre: Skipped 1 previous similar message [16847.369177] do_IRQ: 7 callbacks suppressed [16847.369257] do_IRQ: 0.41 No irq handler for vector [16847.379260] do_IRQ: 0.41 No irq handler for vector [16847.384785] do_IRQ: 0.41 No irq handler for vector [16847.391239] do_IRQ: 0.41 No irq handler for vector [16847.396799] do_IRQ: 0.41 No irq handler for vector [16847.399933] do_IRQ: 0.41 No irq handler for vector [16847.403020] do_IRQ: 0.41 No irq handler for vector [16847.412139] do_IRQ: 0.41 No irq handler for vector [16847.417693] do_IRQ: 0.41 No irq handler for vector [16847.427166] do_IRQ: 0.41 No irq handler for vector [16847.818145] Lustre: lustre-MDT0000: Recovery over after 0:09, of 2 clients 2 recovered and 0 were evicted. [16847.865521] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:24131 to 0x2c0000403:24161) [16847.868530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:30433 to 0x280000403:30497) [16848.067310] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:12295 to 0x280000400:12321) [16848.070054] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:12102 to 0x2c0000400:12129) [16870.438343] Lustre: lustre-MDD0001: changelog off [16870.441604] do_IRQ: 6 callbacks suppressed [16870.441634] do_IRQ: 0.41 No irq handler for vector [16870.444605] Lustre: Skipped 3 previous similar messages [16870.458549] do_IRQ: 0.41 No irq handler for vector [16870.462209] do_IRQ: 0.41 No irq handler for vector [16870.468942] do_IRQ: 0.41 No irq handler for vector [16870.472782] do_IRQ: 0.41 No irq handler for vector [16870.484224] do_IRQ: 0.41 No irq handler for vector [16870.489547] do_IRQ: 0.41 No irq handler for vector [16870.496836] do_IRQ: 0.41 No irq handler for vector [16870.503111] do_IRQ: 0.41 No irq handler for vector [16870.506135] do_IRQ: 0.41 No irq handler for vector [16875.825785] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 04:33:04 (1790757184) [16875.848481] do_IRQ: 1 callbacks suppressed [16875.848492] do_IRQ: 0.41 No irq handler for vector [16875.865458] do_IRQ: 0.41 No irq handler for vector [16875.870476] do_IRQ: 0.41 No irq handler for vector [16875.878193] do_IRQ: 0.41 No irq handler for vector [16875.883421] do_IRQ: 0.41 No irq handler for vector [16875.889274] do_IRQ: 0.41 No irq handler for vector [16875.893861] do_IRQ: 0.41 No irq handler for vector [16875.897113] do_IRQ: 0.41 No irq handler for vector [16875.904701] do_IRQ: 0.41 No irq handler for vector [16875.907927] do_IRQ: 0.41 No irq handler for vector [16887.585426] LustreError: 229868:0:(mdd_device.c:2067:mdd_changelog_user_purge()) cfs_race id 1315 sleeping [16887.599424] do_IRQ: 1 callbacks suppressed [16887.599434] do_IRQ: 0.41 No irq handler for vector [16887.609032] do_IRQ: 0.41 No irq handler for vector [16887.613193] do_IRQ: 0.41 No irq handler for vector [16887.619249] do_IRQ: 0.41 No irq handler for vector [16887.624175] do_IRQ: 0.41 No irq handler for vector [16887.628907] do_IRQ: 0.41 No irq handler for vector [16887.632356] do_IRQ: 0.41 No irq handler for vector [16887.634502] do_IRQ: 0.41 No irq handler for vector [16887.636796] do_IRQ: 0.41 No irq handler for vector [16887.638444] do_IRQ: 0.41 No irq handler for vector [16890.282446] LustreError: 230001:0:(mdd_device.c:1782:mdd_changelog_user_register()) cfs_fail_race id 1315 waking [16890.292482] LustreError: 229868:0:(mdd_device.c:2067:mdd_changelog_user_purge()) cfs_fail_race id 1315 awake: rc=2317 [16892.613928] LustreError: 230202:0:(mdd_device.c:2067:mdd_changelog_user_purge()) cfs_fail_race id 1315 waking [16892.622900] do_IRQ: 3 callbacks suppressed [16892.622907] do_IRQ: 0.41 No irq handler for vector [16892.631972] do_IRQ: 0.41 No irq handler for vector [16892.633718] do_IRQ: 0.41 No irq handler for vector [16892.642423] do_IRQ: 0.41 No irq handler for vector [16892.647189] do_IRQ: 0.41 No irq handler for vector [16892.652133] do_IRQ: 0.41 No irq handler for vector [16892.664455] do_IRQ: 0.41 No irq handler for vector [16892.671485] do_IRQ: 0.41 No irq handler for vector [16892.681711] do_IRQ: 0.41 No irq handler for vector [16892.689849] do_IRQ: 0.41 No irq handler for vector [16916.542777] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 04:33:45 (1790757225) [16916.552560] do_IRQ: 1 callbacks suppressed [16916.552568] do_IRQ: 0.41 No irq handler for vector [16916.559434] do_IRQ: 0.41 No irq handler for vector [16916.566411] do_IRQ: 0.41 No irq handler for vector [16916.575733] do_IRQ: 0.41 No irq handler for vector [16916.583565] do_IRQ: 0.41 No irq handler for vector [16916.585848] do_IRQ: 0.41 No irq handler for vector [16916.587514] do_IRQ: 0.41 No irq handler for vector [16916.590820] do_IRQ: 0.41 No irq handler for vector [16916.593199] do_IRQ: 0.41 No irq handler for vector [16916.595249] do_IRQ: 0.41 No irq handler for vector [16939.917322] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 04:34:08 (1790757248) [16939.929245] do_IRQ: 1 callbacks suppressed [16939.929264] do_IRQ: 0.41 No irq handler for vector [16939.935711] do_IRQ: 0.41 No irq handler for vector [16939.937480] do_IRQ: 0.41 No irq handler for vector [16939.951610] do_IRQ: 0.41 No irq handler for vector [16939.957778] do_IRQ: 0.41 No irq handler for vector [16939.962664] do_IRQ: 0.41 No irq handler for vector [16939.965828] do_IRQ: 0.41 No irq handler for vector [16939.969607] do_IRQ: 0.41 No irq handler for vector [16939.976065] do_IRQ: 0.41 No irq handler for vector [16939.981724] do_IRQ: 0.41 No irq handler for vector [16948.044500] LustreError: 227167:0:(mdd_dir.c:1050:mdd_changelog_store()) cfs_fail_timeout id 15d sleeping for 3000ms [16948.050193] do_IRQ: 1 callbacks suppressed [16948.050217] do_IRQ: 0.41 No irq handler for vector [16948.064332] do_IRQ: 0.41 No irq handler for vector [16948.069645] do_IRQ: 0.41 No irq handler for vector [16948.076526] do_IRQ: 0.41 No irq handler for vector [16948.079431] do_IRQ: 0.41 No irq handler for vector [16948.084155] do_IRQ: 0.41 No irq handler for vector [16948.090675] do_IRQ: 0.41 No irq handler for vector [16948.097965] do_IRQ: 0.41 No irq handler for vector [16948.101691] do_IRQ: 0.41 No irq handler for vector [16948.111299] do_IRQ: 0.41 No irq handler for vector [16951.064507] LustreError: 227167:0:(mdd_dir.c:1050:mdd_changelog_store()) cfs_fail_timeout id 15d awake [16967.635713] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 04:34:36 (1790757276) [16967.660434] do_IRQ: 2 callbacks suppressed [16967.660441] do_IRQ: 0.41 No irq handler for vector [16967.680567] do_IRQ: 0.41 No irq handler for vector [16967.690890] do_IRQ: 0.41 No irq handler for vector [16967.700639] do_IRQ: 0.41 No irq handler for vector [16967.709929] do_IRQ: 0.41 No irq handler for vector [16967.716647] do_IRQ: 0.41 No irq handler for vector [16967.720268] do_IRQ: 0.41 No irq handler for vector [16967.722822] do_IRQ: 0.41 No irq handler for vector [16967.729815] do_IRQ: 0.41 No irq handler for vector [16967.740213] do_IRQ: 0.41 No irq handler for vector [16970.747748] Lustre: lustre-MDD0000: changelog on [16970.756357] Lustre: Skipped 9 previous similar messages [16991.052383] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 04:34:59 (1790757299) [16991.061010] do_IRQ: 3 callbacks suppressed [16991.061021] do_IRQ: 0.41 No irq handler for vector [16991.078580] do_IRQ: 0.41 No irq handler for vector [16991.081381] do_IRQ: 0.41 No irq handler for vector [16991.084714] do_IRQ: 0.41 No irq handler for vector [16991.087925] do_IRQ: 0.41 No irq handler for vector [16991.092010] do_IRQ: 0.41 No irq handler for vector [16991.095059] do_IRQ: 0.41 No irq handler for vector [16991.098773] do_IRQ: 0.41 No irq handler for vector [16991.101546] do_IRQ: 0.41 No irq handler for vector [16991.106276] do_IRQ: 0.41 No irq handler for vector [17005.874870] LustreError: 227166:0:(mdd_device.c:400:llog_changelog_cancel_cb()) cfs_race id 15f sleeping [17005.881758] do_IRQ: 1 callbacks suppressed [17005.881765] do_IRQ: 0.41 No irq handler for vector [17005.887523] do_IRQ: 0.41 No irq handler for vector [17005.892383] do_IRQ: 0.41 No irq handler for vector [17005.895717] do_IRQ: 0.41 No irq handler for vector [17005.897807] do_IRQ: 0.41 No irq handler for vector [17005.900262] do_IRQ: 0.41 No irq handler for vector [17005.902228] do_IRQ: 0.41 No irq handler for vector [17005.906029] do_IRQ: 0.41 No irq handler for vector [17005.908297] do_IRQ: 0.41 No irq handler for vector [17005.911010] do_IRQ: 0.41 No irq handler for vector [17007.867978] LustreError: 227703:0:(mdd_device.c:400:llog_changelog_cancel_cb()) cfs_fail_race id 15f waking [17007.884071] LustreError: 227703:0:(mdd_device.c:400:llog_changelog_cancel_cb()) Skipped 1 previous similar message [17007.891374] LustreError: 227166:0:(mdd_device.c:400:llog_changelog_cancel_cb()) cfs_fail_race id 15f awake: rc=2990 [17019.695799] Lustre: lustre-MDD0001: changelog off [17019.698010] do_IRQ: 4 callbacks suppressed [17019.698019] do_IRQ: 0.41 No irq handler for vector [17019.700436] Lustre: Skipped 11 previous similar messages [17019.723138] do_IRQ: 0.41 No irq handler for vector [17019.739871] do_IRQ: 0.41 No irq handler for vector [17019.744344] do_IRQ: 0.41 No irq handler for vector [17019.754827] do_IRQ: 0.41 No irq handler for vector [17019.760378] do_IRQ: 0.41 No irq handler for vector [17019.763302] do_IRQ: 0.41 No irq handler for vector [17019.772805] do_IRQ: 0.41 No irq handler for vector [17019.777323] do_IRQ: 0.41 No irq handler for vector [17019.780416] do_IRQ: 0.41 No irq handler for vector [17024.500451] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 04:35:33 (1790757333) [18890.775614] LustreError: 232535:0:(mdd_device.c:414:llog_changelog_cancel_cb()) cfs_race id 16c sleeping [18890.787846] do_IRQ: 2 callbacks suppressed [18890.787855] do_IRQ: 0.41 No irq handler for vector [18890.800339] do_IRQ: 0.41 No irq handler for vector [18890.803176] do_IRQ: 0.41 No irq handler for vector [18890.818187] do_IRQ: 0.41 No irq handler for vector [18890.825023] do_IRQ: 0.41 No irq handler for vector [18890.830125] do_IRQ: 0.41 No irq handler for vector [18890.834335] do_IRQ: 0.41 No irq handler for vector [18890.837240] do_IRQ: 0.41 No irq handler for vector [18890.840444] do_IRQ: 0.41 No irq handler for vector [18890.843231] do_IRQ: 0.41 No irq handler for vector [18892.804421] LustreError: 227168:0:(llog_osd.c:1130:llog_osd_next_block()) cfs_fail_race id 16c waking [18892.818224] LustreError: 232535:0:(mdd_device.c:414:llog_changelog_cancel_cb()) cfs_fail_race id 16c awake: rc=2969 [18903.576717] Lustre: lustre-MDD0001: changelog off [18903.581862] do_IRQ: 3 callbacks suppressed [18903.581882] do_IRQ: 0.41 No irq handler for vector [18903.584522] Lustre: Skipped 1 previous similar message [18903.613572] do_IRQ: 0.41 No irq handler for vector [18903.623657] do_IRQ: 0.41 No irq handler for vector [18903.629729] do_IRQ: 0.41 No irq handler for vector [18903.639444] do_IRQ: 0.41 No irq handler for vector [18903.643492] do_IRQ: 0.41 No irq handler for vector [18903.649634] do_IRQ: 0.41 No irq handler for vector [18903.657018] do_IRQ: 0.41 No irq handler for vector [18903.662751] do_IRQ: 0.41 No irq handler for vector [18903.666854] do_IRQ: 0.41 No irq handler for vector [18909.085627] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 05:06:58 (1790759218) [18909.096409] do_IRQ: 1 callbacks suppressed [18909.096421] do_IRQ: 0.41 No irq handler for vector [18909.102453] do_IRQ: 0.41 No irq handler for vector [18909.106522] do_IRQ: 0.41 No irq handler for vector [18909.109514] do_IRQ: 0.41 No irq handler for vector [18909.112692] do_IRQ: 0.41 No irq handler for vector [18909.115695] do_IRQ: 0.41 No irq handler for vector [18909.118185] do_IRQ: 0.41 No irq handler for vector [18909.121055] do_IRQ: 0.41 No irq handler for vector [18909.123315] do_IRQ: 0.41 No irq handler for vector [18909.125406] do_IRQ: 0.41 No irq handler for vector [18911.914344] Lustre: lustre-MDD0000: changelog on [18911.922199] Lustre: Skipped 5 previous similar messages [18915.593139] LustreError: 238078:0:(mdd_device.c:1736:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [18915.602857] do_IRQ: 4 callbacks suppressed [18915.602867] do_IRQ: 0.41 No irq handler for vector [18915.605962] do_IRQ: 0.41 No irq handler for vector [18915.610061] do_IRQ: 0.41 No irq handler for vector [18915.616763] do_IRQ: 0.41 No irq handler for vector [18915.618476] do_IRQ: 0.41 No irq handler for vector [18915.631385] do_IRQ: 0.41 No irq handler for vector [18915.636768] do_IRQ: 0.41 No irq handler for vector [18915.639239] do_IRQ: 0.41 No irq handler for vector [18915.641274] do_IRQ: 0.41 No irq handler for vector [18915.644843] do_IRQ: 0.41 No irq handler for vector [18916.519446] Lustre: 238127:0:(mdd_device.c:1753:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [18917.341527] LustreError: 238175:0:(mdd_device.c:1745:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [18942.964370] Lustre: lustre-MDD0000: changelog off [18942.968484] do_IRQ: 4 callbacks suppressed [18942.968497] do_IRQ: 0.41 No irq handler for vector [18942.972156] Lustre: Skipped 1 previous similar message [18942.989626] do_IRQ: 0.41 No irq handler for vector [18942.993720] do_IRQ: 0.41 No irq handler for vector [18942.995496] do_IRQ: 0.41 No irq handler for vector [18943.001453] do_IRQ: 0.41 No irq handler for vector [18943.004419] do_IRQ: 0.41 No irq handler for vector [18943.007396] do_IRQ: 0.41 No irq handler for vector [18943.009750] do_IRQ: 0.41 No irq handler for vector [18943.012546] do_IRQ: 0.41 No irq handler for vector [18943.015432] do_IRQ: 0.41 No irq handler for vector [18950.038438] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 05:07:39 (1790759259) [18950.047764] do_IRQ: 1 callbacks suppressed [18950.047776] do_IRQ: 0.41 No irq handler for vector [18950.054513] do_IRQ: 0.41 No irq handler for vector [18950.057397] do_IRQ: 0.41 No irq handler for vector [18950.059959] do_IRQ: 0.41 No irq handler for vector [18950.063177] do_IRQ: 0.41 No irq handler for vector [18950.065994] do_IRQ: 0.41 No irq handler for vector [18950.068850] do_IRQ: 0.41 No irq handler for vector [18950.072911] do_IRQ: 0.41 No irq handler for vector [18950.074734] do_IRQ: 0.41 No irq handler for vector [18950.078507] do_IRQ: 0.41 No irq handler for vector [18952.744490] Lustre: lustre-MDD0000: changelog on [18952.746796] Lustre: Skipped 1 previous similar message [18959.434500] Lustre: Failing over lustre-MDT0000 [18959.441908] do_IRQ: 3 callbacks suppressed [18959.441916] do_IRQ: 0.41 No irq handler for vector [18959.459568] do_IRQ: 0.41 No irq handler for vector [18959.468354] do_IRQ: 0.41 No irq handler for vector [18959.474705] do_IRQ: 0.41 No irq handler for vector [18959.481473] do_IRQ: 0.41 No irq handler for vector [18959.489906] do_IRQ: 0.41 No irq handler for vector [18959.492442] do_IRQ: 0.41 No irq handler for vector [18959.494605] do_IRQ: 0.41 No irq handler for vector [18959.497143] do_IRQ: 0.41 No irq handler for vector [18959.499010] do_IRQ: 0.41 No irq handler for vector [18959.794506] Lustre: server umount lustre-MDT0000 complete [18961.897520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18961.902238] LustreError: 232535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18961.909318] Lustre: Skipped 3 previous similar messages [18961.926637] LustreError: 232535:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [18963.163055] LustreError: 227703:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [18967.017890] LustreError: 229004:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18967.038388] do_IRQ: 7 callbacks suppressed [18967.038399] do_IRQ: 0.41 No irq handler for vector [18967.041688] do_IRQ: 0.41 No irq handler for vector [18967.045433] do_IRQ: 0.41 No irq handler for vector [18967.048930] do_IRQ: 0.41 No irq handler for vector [18967.051641] do_IRQ: 0.41 No irq handler for vector [18967.053512] do_IRQ: 0.41 No irq handler for vector [18967.058113] do_IRQ: 0.41 No irq handler for vector [18967.062541] do_IRQ: 0.41 No irq handler for vector [18967.065937] do_IRQ: 0.41 No irq handler for vector [18967.072248] do_IRQ: 0.41 No irq handler for vector [18967.078789] LustreError: 229004:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [18968.348947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18968.562406] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18968.973560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18969.134410] Lustre: 240508:0:(mdd_device.c:628:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 90157 to index 18446744073709551615, being cleared now [18969.164433] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [18973.366894] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [18973.373271] do_IRQ: 8 callbacks suppressed [18973.373288] do_IRQ: 0.41 No irq handler for vector [18973.381389] do_IRQ: 0.41 No irq handler for vector [18973.387603] do_IRQ: 0.41 No irq handler for vector [18973.395863] do_IRQ: 0.41 No irq handler for vector [18973.397497] do_IRQ: 0.41 No irq handler for vector [18973.405817] do_IRQ: 0.41 No irq handler for vector [18973.410227] do_IRQ: 0.41 No irq handler for vector [18973.413281] do_IRQ: 0.41 No irq handler for vector [18973.416448] do_IRQ: 0.41 No irq handler for vector [18973.419895] do_IRQ: 0.41 No irq handler for vector [18973.771624] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [18974.195639] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [18974.208563] Lustre: Skipped 3 previous similar messages [18974.275986] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [18974.289137] Lustre: Skipped 1 previous similar message [18974.315955] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:24163 to 0x2c0000403:24193) [18974.321372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:30499 to 0x280000403:30529) [18991.353822] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 05:08:19 (1790759299) [18991.392583] do_IRQ: 8 callbacks suppressed [18991.392591] do_IRQ: 0.41 No irq handler for vector [18991.395147] do_IRQ: 0.41 No irq handler for vector [18991.409863] do_IRQ: 0.41 No irq handler for vector [18991.423786] do_IRQ: 0.41 No irq handler for vector [18991.426414] do_IRQ: 0.41 No irq handler for vector [18991.429421] do_IRQ: 0.41 No irq handler for vector [18991.431203] do_IRQ: 0.41 No irq handler for vector [18991.437417] do_IRQ: 0.41 No irq handler for vector [18991.444777] do_IRQ: 0.41 No irq handler for vector [18991.450259] do_IRQ: 0.41 No irq handler for vector [19004.504534] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records * time ========================================================== 05:08:32 (1790759312) [19004.523654] do_IRQ: 2 callbacks suppressed [19004.523661] do_IRQ: 0.41 No irq handler for vector [19004.534222] do_IRQ: 0.41 No irq handler for vector [19004.541450] do_IRQ: 0.41 No irq handler for vector [19004.548710] do_IRQ: 0.41 No irq handler for vector [19004.553487] do_IRQ: 0.41 No irq handler for vector [19004.564530] do_IRQ: 0.41 No irq handler for vector [19004.570471] do_IRQ: 0.41 No irq handler for vector [19004.580481] do_IRQ: 0.41 No irq handler for vector [19004.584016] do_IRQ: 0.41 No irq handler for vector [19004.593722] do_IRQ: 0.41 No irq handler for vector [19026.022943] Lustre: 229004:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [19026.031154] Lustre: 229004:0:(mdd_dir.c:1082:mdd_changelog_store()) Skipped 1 previous similar message [19026.031259] do_IRQ: 1 callbacks suppressed [19026.031265] do_IRQ: 0.41 No irq handler for vector [19026.054776] do_IRQ: 0.41 No irq handler for vector [19026.059538] do_IRQ: 0.41 No irq handler for vector [19026.062698] do_IRQ: 0.41 No irq handler for vector [19026.070364] do_IRQ: 0.41 No irq handler for vector [19026.073440] do_IRQ: 0.41 No irq handler for vector [19026.079316] do_IRQ: 0.41 No irq handler for vector [19026.085137] do_IRQ: 0.41 No irq handler for vector [19026.090027] do_IRQ: 0.41 No irq handler for vector [19026.093101] do_IRQ: 0.41 No irq handler for vector [19026.103103] Lustre: 242509:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl2 idle for 864016s with 500000004 unprocessed records [19026.136895] Lustre: 242509:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [19026.148745] Lustre: lustre-MDD0000: changelog off [19026.159477] Lustre: Skipped 3 previous similar messages [19046.568500] Lustre: DEBUG MARKER: == sanity test 160t: changelog garbage collect on lack of space ========================================================== 05:09:15 (1790759355) [19046.574854] do_IRQ: 5 callbacks suppressed [19046.574865] do_IRQ: 0.41 No irq handler for vector [19046.578802] do_IRQ: 0.41 No irq handler for vector [19046.584494] do_IRQ: 0.41 No irq handler for vector [19046.587271] do_IRQ: 0.41 No irq handler for vector [19046.589298] do_IRQ: 0.41 No irq handler for vector [19046.592061] do_IRQ: 0.41 No irq handler for vector [19046.594422] do_IRQ: 0.41 No irq handler for vector [19046.596456] do_IRQ: 0.41 No irq handler for vector [19046.599094] do_IRQ: 0.41 No irq handler for vector [19046.601287] do_IRQ: 0.41 No irq handler for vector [19049.498285] Lustre: lustre-MDD0000: changelog on [19049.501120] Lustre: Skipped 4 previous similar messages [19111.131506] Lustre: *** cfs_fail_loc=18c, val=1210732*** [19111.141091] Lustre: Skipped 1 previous similar message [19111.141188] do_IRQ: 3 callbacks suppressed [19111.141191] do_IRQ: 0.41 No irq handler for vector [19111.147124] Lustre: 229004:0:(mdd_dir.c:1082:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [19111.147136] Lustre: 229004:0:(mdd_dir.c:1082:mdd_changelog_store()) Skipped 1 previous similar message [19111.154129] Lustre: 244510:0:(mdd_dir.c:966:mdd_changelog_is_space_safe()) lustre-MDD0000: changelog size 1MB with 1MB space limit [19111.154435] Lustre: 244510:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl3-user1 idle for 62s with 7506 unprocessed records [19111.154440] Lustre: 244510:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [19111.230614] do_IRQ: 0.41 No irq handler for vector [19111.233413] do_IRQ: 0.41 No irq handler for vector [19111.235404] do_IRQ: 0.41 No irq handler for vector [19111.239976] do_IRQ: 0.41 No irq handler for vector [19111.242423] do_IRQ: 0.41 No irq handler for vector [19111.244543] do_IRQ: 0.41 No irq handler for vector [19111.255946] do_IRQ: 0.41 No irq handler for vector [19111.259653] do_IRQ: 0.41 No irq handler for vector [19111.262688] do_IRQ: 0.41 No irq handler for vector [19163.577804] Lustre: lustre-MDD0001: changelog off [19163.584837] do_IRQ: 1 callbacks suppressed [19163.584845] do_IRQ: 0.41 No irq handler for vector [19163.589558] do_IRQ: 0.41 No irq handler for vector [19163.593386] do_IRQ: 0.41 No irq handler for vector [19163.595345] do_IRQ: 0.41 No irq handler for vector [19163.596512] do_IRQ: 0.41 No irq handler for vector [19163.603696] do_IRQ: 0.41 No irq handler for vector [19163.610604] do_IRQ: 0.41 No irq handler for vector [19163.615142] do_IRQ: 0.41 No irq handler for vector [19163.626383] do_IRQ: 0.41 No irq handler for vector [19163.632779] do_IRQ: 0.41 No irq handler for vector [19163.636899] Lustre: Skipped 2 previous similar messages [19170.100652] Lustre: DEBUG MARKER: == sanity test 160u: changelog rename record type name and sname strings are correct ========================================================== 05:11:18 (1790759478) [19170.114129] do_IRQ: 3 callbacks suppressed [19170.114142] do_IRQ: 0.41 No irq handler for vector [19170.122778] do_IRQ: 0.41 No irq handler for vector [19170.134188] do_IRQ: 0.41 No irq handler for vector [19170.140253] do_IRQ: 0.41 No irq handler for vector [19170.147183] do_IRQ: 0.41 No irq handler for vector [19170.151754] do_IRQ: 0.41 No irq handler for vector [19170.155229] do_IRQ: 0.41 No irq handler for vector [19170.159995] do_IRQ: 0.41 No irq handler for vector [19170.162421] do_IRQ: 0.41 No irq handler for vector [19170.164313] do_IRQ: 0.41 No irq handler for vector [19178.859740] Lustre: lustre-MDD0001: changelog on [19178.863180] do_IRQ: 1 callbacks suppressed [19178.863193] do_IRQ: 0.41 No irq handler for vector [19178.865744] Lustre: Skipped 2 previous similar messages [19178.877870] do_IRQ: 0.41 No irq handler for vector [19178.881668] do_IRQ: 0.41 No irq handler for vector [19178.885536] do_IRQ: 0.41 No irq handler for vector [19178.888715] do_IRQ: 0.41 No irq handler for vector [19178.891608] do_IRQ: 0.41 No irq handler for vector [19178.894892] do_IRQ: 0.41 No irq handler for vector [19178.899202] do_IRQ: 0.41 No irq handler for vector [19178.903173] do_IRQ: 0.41 No irq handler for vector [19178.906616] do_IRQ: 0.41 No irq handler for vector [19194.966869] Lustre: DEBUG MARKER: == sanity test 160v: setattr preserves mtime update despite inter-client clock skew ========================================================== 05:11:43 (1790759503) [19194.981918] do_IRQ: 1 callbacks suppressed [19194.981934] do_IRQ: 0.41 No irq handler for vector [19194.985265] do_IRQ: 0.41 No irq handler for vector [19194.992700] do_IRQ: 0.41 No irq handler for vector [19195.000928] do_IRQ: 0.41 No irq handler for vector [19195.003287] do_IRQ: 0.41 No irq handler for vector [19195.006592] do_IRQ: 0.41 No irq handler for vector [19195.013510] do_IRQ: 0.41 No irq handler for vector [19195.021826] do_IRQ: 0.41 No irq handler for vector [19195.028925] do_IRQ: 0.41 No irq handler for vector [19195.033408] do_IRQ: 0.41 No irq handler for vector [19196.883131] Lustre: DEBUG MARKER: SKIP: sanity test_160v need >= 2 clients [19198.692904] Lustre: DEBUG MARKER: == sanity test 160w: lfs changelog --user --mask ========= 05:11:47 (1790759507) [19232.893292] Lustre: DEBUG MARKER: == sanity test 160x: changelog users do not disappear ==== 05:12:21 (1790759541) [19232.900957] do_IRQ: 3 callbacks suppressed [19232.900970] do_IRQ: 0.41 No irq handler for vector [19232.919900] do_IRQ: 0.41 No irq handler for vector [19232.925757] do_IRQ: 0.41 No irq handler for vector [19232.934627] do_IRQ: 0.41 No irq handler for vector [19232.939751] do_IRQ: 0.41 No irq handler for vector [19232.949903] do_IRQ: 0.41 No irq handler for vector [19232.956837] do_IRQ: 0.41 No irq handler for vector [19232.965039] do_IRQ: 0.41 No irq handler for vector [19232.966722] do_IRQ: 0.41 No irq handler for vector [19232.984920] do_IRQ: 0.41 No irq handler for vector [19296.239955] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 05:13:25 (1790759605) [19296.246795] do_IRQ: 1 callbacks suppressed [19296.246804] do_IRQ: 0.41 No irq handler for vector [19296.252458] do_IRQ: 0.41 No irq handler for vector [19296.256204] do_IRQ: 0.41 No irq handler for vector [19296.262065] do_IRQ: 0.41 No irq handler for vector [19296.267230] do_IRQ: 0.41 No irq handler for vector [19296.273509] do_IRQ: 0.41 No irq handler for vector [19296.278402] do_IRQ: 0.41 No irq handler for vector [19296.282454] do_IRQ: 0.41 No irq handler for vector [19296.292466] do_IRQ: 0.41 No irq handler for vector [19296.298492] do_IRQ: 0.41 No irq handler for vector [19331.777366] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 05:14:00 (1790759640) [19331.793161] do_IRQ: 1 callbacks suppressed [19331.793172] do_IRQ: 0.41 No irq handler for vector [19331.797715] do_IRQ: 0.41 No irq handler for vector [19331.806125] do_IRQ: 0.41 No irq handler for vector [19331.813281] do_IRQ: 0.41 No irq handler for vector [19331.820235] do_IRQ: 0.41 No irq handler for vector [19331.822706] do_IRQ: 0.41 No irq handler for vector [19331.825917] do_IRQ: 0.41 No irq handler for vector [19331.828979] do_IRQ: 0.41 No irq handler for vector [19331.831114] do_IRQ: 0.41 No irq handler for vector [19331.833557] do_IRQ: 0.41 No irq handler for vector [19371.957999] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 05:14:40 (1790759680) [19371.967685] do_IRQ: 2 callbacks suppressed [19371.967695] do_IRQ: 0.41 No irq handler for vector [19371.969563] do_IRQ: 0.41 No irq handler for vector [19371.972025] do_IRQ: 0.41 No irq handler for vector [19371.974187] do_IRQ: 0.41 No irq handler for vector [19371.976253] do_IRQ: 0.41 No irq handler for vector [19371.978377] do_IRQ: 0.41 No irq handler for vector [19371.980045] do_IRQ: 0.41 No irq handler for vector [19371.982141] do_IRQ: 0.41 No irq handler for vector [19371.984941] do_IRQ: 0.41 No irq handler for vector [19371.987602] do_IRQ: 0.41 No irq handler for vector [19394.461828] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 05:15:03 (1790759703) [19394.469445] do_IRQ: 2 callbacks suppressed [19394.469458] do_IRQ: 0.41 No irq handler for vector [19394.475651] do_IRQ: 0.41 No irq handler for vector [19394.478310] do_IRQ: 0.41 No irq handler for vector [19394.480299] do_IRQ: 0.41 No irq handler for vector [19394.482980] do_IRQ: 0.41 No irq handler for vector [19394.484637] do_IRQ: 0.41 No irq handler for vector [19394.487451] do_IRQ: 0.41 No irq handler for vector [19394.490287] do_IRQ: 0.41 No irq handler for vector [19394.493434] do_IRQ: 0.41 No irq handler for vector [19394.497338] do_IRQ: 0.41 No irq handler for vector [19414.967243] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 05:15:24 (1790759724) [19414.973552] do_IRQ: 1 callbacks suppressed [19414.973564] do_IRQ: 0.41 No irq handler for vector [19414.976264] do_IRQ: 0.41 No irq handler for vector [19414.983925] do_IRQ: 0.41 No irq handler for vector [19414.994657] do_IRQ: 0.41 No irq handler for vector [19414.999813] do_IRQ: 0.41 No irq handler for vector [19415.003183] do_IRQ: 0.41 No irq handler for vector [19415.006287] do_IRQ: 0.41 No irq handler for vector [19415.010721] do_IRQ: 0.41 No irq handler for vector [19415.012569] do_IRQ: 0.41 No irq handler for vector [19415.018876] do_IRQ: 0.41 No irq handler for vector [19423.226679] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 05:15:32 (1790759732) [19423.234384] do_IRQ: 1 callbacks suppressed [19423.234394] do_IRQ: 0.41 No irq handler for vector [19423.238775] do_IRQ: 0.41 No irq handler for vector [19423.242923] do_IRQ: 0.41 No irq handler for vector [19423.248632] do_IRQ: 0.41 No irq handler for vector [19423.269083] do_IRQ: 0.41 No irq handler for vector [19423.274863] do_IRQ: 0.41 No irq handler for vector [19423.284172] do_IRQ: 0.41 No irq handler for vector [19423.289658] do_IRQ: 0.41 No irq handler for vector [19423.297586] do_IRQ: 0.41 No irq handler for vector [19423.306263] do_IRQ: 0.41 No irq handler for vector [19431.213883] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 05:15:40 (1790759740) [19431.222859] do_IRQ: 1 callbacks suppressed [19431.222871] do_IRQ: 0.41 No irq handler for vector [19431.230114] do_IRQ: 0.41 No irq handler for vector [19431.235288] do_IRQ: 0.41 No irq handler for vector [19431.241626] do_IRQ: 0.41 No irq handler for vector [19431.248417] do_IRQ: 0.41 No irq handler for vector [19431.252451] do_IRQ: 0.41 No irq handler for vector [19431.254895] do_IRQ: 0.41 No irq handler for vector [19431.257508] do_IRQ: 0.41 No irq handler for vector [19431.259713] do_IRQ: 0.41 No irq handler for vector [19431.262212] do_IRQ: 0.41 No irq handler for vector [19536.167383] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 05:17:24 (1790759844) [19536.174310] do_IRQ: 1 callbacks suppressed [19536.174321] do_IRQ: 0.41 No irq handler for vector [19536.184263] do_IRQ: 0.41 No irq handler for vector [19536.189498] do_IRQ: 0.41 No irq handler for vector [19536.193912] do_IRQ: 0.41 No irq handler for vector [19536.198780] do_IRQ: 0.41 No irq handler for vector [19536.203214] do_IRQ: 0.41 No irq handler for vector [19536.207117] do_IRQ: 0.41 No irq handler for vector [19536.212367] do_IRQ: 0.41 No irq handler for vector [19536.214648] do_IRQ: 0.41 No irq handler for vector [19536.216890] do_IRQ: 0.41 No irq handler for vector [19544.520170] Lustre: Failing over lustre-OST0000 [19544.523783] do_IRQ: 1 callbacks suppressed [19544.523791] do_IRQ: 0.41 No irq handler for vector [19544.526947] do_IRQ: 0.41 No irq handler for vector [19544.528577] do_IRQ: 0.41 No irq handler for vector [19544.532271] do_IRQ: 0.41 No irq handler for vector [19544.534632] do_IRQ: 0.41 No irq handler for vector [19544.537074] do_IRQ: 0.41 No irq handler for vector [19544.541760] do_IRQ: 0.41 No irq handler for vector [19544.544162] do_IRQ: 0.41 No irq handler for vector [19544.548736] do_IRQ: 0.41 No irq handler for vector [19544.553724] do_IRQ: 0.41 No irq handler for vector [19544.809807] Lustre: server umount lustre-OST0000 complete [19546.595936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [19546.616475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19546.632676] Lustre: Skipped 1 previous similar message [19546.639232] LustreError: 204277:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19546.657899] LustreError: 204277:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [19547.628888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19557.052120] LustreError: 123680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [19557.073978] do_IRQ: 8 callbacks suppressed [19557.073989] do_IRQ: 0.41 No irq handler for vector [19557.076827] do_IRQ: 0.41 No irq handler for vector [19557.081119] do_IRQ: 0.41 No irq handler for vector [19557.085781] do_IRQ: 0.41 No irq handler for vector [19557.089477] do_IRQ: 0.41 No irq handler for vector [19557.092341] do_IRQ: 0.41 No irq handler for vector [19557.095831] do_IRQ: 0.41 No irq handler for vector [19557.104699] do_IRQ: 0.41 No irq handler for vector [19557.109233] do_IRQ: 0.41 No irq handler for vector [19557.115556] do_IRQ: 0.41 No irq handler for vector [19557.126140] LustreError: 123680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [19561.712264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19561.995287] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [19562.013436] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [19562.172295] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [19562.179866] do_IRQ: 6 callbacks suppressed [19562.179875] do_IRQ: 0.41 No irq handler for vector [19562.184973] do_IRQ: 0.41 No irq handler for vector [19562.187288] do_IRQ: 0.41 No irq handler for vector [19562.194642] do_IRQ: 0.41 No irq handler for vector [19562.198018] do_IRQ: 0.41 No irq handler for vector [19562.200723] do_IRQ: 0.41 No irq handler for vector [19562.203339] do_IRQ: 0.41 No irq handler for vector [19562.205444] do_IRQ: 0.41 No irq handler for vector [19562.210516] do_IRQ: 0.41 No irq handler for vector [19562.212419] do_IRQ: 0.41 No irq handler for vector [19563.470947] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [19563.475117] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [19563.485546] Lustre: Skipped 3 previous similar messages [19566.885524] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [19570.050651] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 05:17:59 (1790759879) [19570.060795] do_IRQ: 6 callbacks suppressed [19570.060808] do_IRQ: 0.41 No irq handler for vector [19570.072914] do_IRQ: 0.41 No irq handler for vector [19570.076329] do_IRQ: 0.41 No irq handler for vector [19570.080428] do_IRQ: 0.41 No irq handler for vector [19570.083383] do_IRQ: 0.41 No irq handler for vector [19570.086179] do_IRQ: 0.41 No irq handler for vector [19570.088482] do_IRQ: 0.41 No irq handler for vector [19570.092726] do_IRQ: 0.41 No irq handler for vector [19570.098916] do_IRQ: 0.41 No irq handler for vector [19570.105886] do_IRQ: 0.41 No irq handler for vector [19601.432927] Lustre: Failing over lustre-OST0000 [19601.439223] do_IRQ: 1 callbacks suppressed [19601.439229] do_IRQ: 0.41 No irq handler for vector [19601.444013] do_IRQ: 0.41 No irq handler for vector [19601.447748] do_IRQ: 0.41 No irq handler for vector [19601.453515] do_IRQ: 0.41 No irq handler for vector [19601.458635] do_IRQ: 0.41 No irq handler for vector [19601.460884] do_IRQ: 0.41 No irq handler for vector [19601.463354] do_IRQ: 0.41 No irq handler for vector [19601.465461] do_IRQ: 0.41 No irq handler for vector [19601.470730] do_IRQ: 0.41 No irq handler for vector [19601.474236] do_IRQ: 0.41 No irq handler for vector [19601.640920] Lustre: server umount lustre-OST0000 complete [19602.922832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19602.950789] Lustre: Skipped 1 previous similar message [19602.952265] LustreError: 123678:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19602.994578] LustreError: 123678:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [19609.436095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19609.445896] do_IRQ: 6 callbacks suppressed [19609.445908] do_IRQ: 0.41 No irq handler for vector [19609.456196] do_IRQ: 0.41 No irq handler for vector [19609.462584] do_IRQ: 0.41 No irq handler for vector [19609.470555] do_IRQ: 0.41 No irq handler for vector [19609.476459] do_IRQ: 0.41 No irq handler for vector [19609.478191] do_IRQ: 0.41 No irq handler for vector [19609.483750] do_IRQ: 0.41 No irq handler for vector [19609.491065] do_IRQ: 0.41 No irq handler for vector [19609.495247] do_IRQ: 0.41 No irq handler for vector [19609.505309] do_IRQ: 0.41 No irq handler for vector [19609.834734] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [19609.854091] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [19611.363075] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [19611.651640] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [19611.651888] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [19611.668747] Lustre: Skipped 1 previous similar message [19615.433853] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [19615.438245] do_IRQ: 7 callbacks suppressed [19615.438255] do_IRQ: 0.41 No irq handler for vector [19615.442453] do_IRQ: 0.41 No irq handler for vector [19615.446263] do_IRQ: 0.41 No irq handler for vector [19615.448979] do_IRQ: 0.41 No irq handler for vector [19615.452620] do_IRQ: 0.41 No irq handler for vector [19615.455473] do_IRQ: 0.41 No irq handler for vector [19615.458818] do_IRQ: 0.41 No irq handler for vector [19615.461335] do_IRQ: 0.41 No irq handler for vector [19615.463934] do_IRQ: 0.41 No irq handler for vector [19615.466601] do_IRQ: 0.41 No irq handler for vector [19618.671600] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 05:18:47 (1790759927) [19644.812848] Lustre: Failing over lustre-OST0000 [19644.818344] do_IRQ: 3 callbacks suppressed [19644.818353] do_IRQ: 0.41 No irq handler for vector [19644.833449] do_IRQ: 0.41 No irq handler for vector [19644.836056] do_IRQ: 0.41 No irq handler for vector [19644.838652] do_IRQ: 0.41 No irq handler for vector [19644.840527] do_IRQ: 0.41 No irq handler for vector [19644.846496] do_IRQ: 0.41 No irq handler for vector [19644.848894] do_IRQ: 0.41 No irq handler for vector [19644.852957] do_IRQ: 0.41 No irq handler for vector [19644.854837] do_IRQ: 0.41 No irq handler for vector [19644.856113] do_IRQ: 0.41 No irq handler for vector [19645.926502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19645.957675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19645.963363] Lustre: Skipped 2 previous similar messages [19646.990653] Lustre: server umount lustre-OST0000 complete [19649.232292] LustreError: 204026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [19649.242318] LustreError: 204026:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [19654.413155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19654.425255] do_IRQ: 7 callbacks suppressed [19654.425265] do_IRQ: 0.41 No irq handler for vector [19654.432803] do_IRQ: 0.41 No irq handler for vector [19654.436105] do_IRQ: 0.41 No irq handler for vector [19654.439950] do_IRQ: 0.41 No irq handler for vector [19654.446173] do_IRQ: 0.41 No irq handler for vector [19654.451535] do_IRQ: 0.41 No irq handler for vector [19654.454110] do_IRQ: 0.41 No irq handler for vector [19654.456515] do_IRQ: 0.41 No irq handler for vector [19654.462102] do_IRQ: 0.41 No irq handler for vector [19654.465259] do_IRQ: 0.41 No irq handler for vector [19654.809039] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [19654.829378] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [19656.270805] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [19656.834830] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [19656.834898] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [19656.854601] Lustre: Skipped 1 previous similar message [19660.574558] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [19660.581165] do_IRQ: 7 callbacks suppressed [19660.581176] do_IRQ: 0.41 No irq handler for vector [19660.583803] do_IRQ: 0.41 No irq handler for vector [19660.587353] do_IRQ: 0.41 No irq handler for vector [19660.591326] do_IRQ: 0.41 No irq handler for vector [19660.594747] do_IRQ: 0.41 No irq handler for vector [19660.601058] do_IRQ: 0.41 No irq handler for vector [19660.605406] do_IRQ: 0.41 No irq handler for vector [19660.609909] do_IRQ: 0.41 No irq handler for vector [19660.617446] do_IRQ: 0.41 No irq handler for vector [19660.623980] do_IRQ: 0.41 No irq handler for vector [19664.016652] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 05:19:33 (1790759973) [19704.971452] Lustre: Failing over lustre-OST0000 [19704.977671] do_IRQ: 3 callbacks suppressed [19704.977679] do_IRQ: 0.41 No irq handler for vector [19704.980489] do_IRQ: 0.41 No irq handler for vector [19704.985319] do_IRQ: 0.41 No irq handler for vector [19704.987410] do_IRQ: 0.41 No irq handler for vector [19704.991309] do_IRQ: 0.41 No irq handler for vector [19704.997741] do_IRQ: 0.41 No irq handler for vector [19705.000551] do_IRQ: 0.41 No irq handler for vector [19705.006447] do_IRQ: 0.41 No irq handler for vector [19705.009560] do_IRQ: 0.41 No irq handler for vector [19705.011977] do_IRQ: 0.41 No irq handler for vector [19705.155774] Lustre: server umount lustre-OST0000 complete [19706.351430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19706.366918] Lustre: Skipped 1 previous similar message [19713.462174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19713.468210] do_IRQ: 5 callbacks suppressed [19713.468220] do_IRQ: 0.41 No irq handler for vector [19713.470641] do_IRQ: 0.41 No irq handler for vector [19713.473151] do_IRQ: 0.41 No irq handler for vector [19713.475762] do_IRQ: 0.41 No irq handler for vector [19713.480462] do_IRQ: 0.41 No irq handler for vector [19713.482894] do_IRQ: 0.41 No irq handler for vector [19713.485422] do_IRQ: 0.41 No irq handler for vector [19713.488030] do_IRQ: 0.41 No irq handler for vector [19713.490639] do_IRQ: 0.41 No irq handler for vector [19713.494704] do_IRQ: 0.41 No irq handler for vector [19713.780571] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [19713.796703] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [19715.742236] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [19715.800352] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [19715.804308] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [19715.812050] Lustre: Skipped 1 previous similar message [19719.866270] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [19719.875934] do_IRQ: 8 callbacks suppressed [19719.875944] do_IRQ: 0.41 No irq handler for vector [19719.880494] do_IRQ: 0.41 No irq handler for vector [19719.883887] do_IRQ: 0.41 No irq handler for vector [19719.886198] do_IRQ: 0.41 No irq handler for vector [19719.889952] do_IRQ: 0.41 No irq handler for vector [19719.893553] do_IRQ: 0.41 No irq handler for vector [19719.895342] do_IRQ: 0.41 No irq handler for vector [19719.900539] do_IRQ: 0.41 No irq handler for vector [19719.904922] do_IRQ: 0.41 No irq handler for vector [19719.907332] do_IRQ: 0.41 No irq handler for vector [19723.184950] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 05:20:32 (1790760032) [19742.865369] Lustre: Failing over lustre-OST0000 [19742.870030] do_IRQ: 3 callbacks suppressed [19742.870039] do_IRQ: 0.41 No irq handler for vector [19742.884749] do_IRQ: 0.41 No irq handler for vector [19742.889437] do_IRQ: 0.41 No irq handler for vector [19742.893357] do_IRQ: 0.41 No irq handler for vector [19742.898688] do_IRQ: 0.41 No irq handler for vector [19742.900688] do_IRQ: 0.41 No irq handler for vector [19742.903363] do_IRQ: 0.41 No irq handler for vector [19742.906062] do_IRQ: 0.41 No irq handler for vector [19742.908462] do_IRQ: 0.41 No irq handler for vector [19742.911322] do_IRQ: 0.41 No irq handler for vector [19743.012803] Lustre: server umount lustre-OST0000 complete [19744.743052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19744.762031] Lustre: Skipped 1 previous similar message [19744.765200] LustreError: 188647:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19744.776149] LustreError: 188647:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [19750.191789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19750.203573] do_IRQ: 6 callbacks suppressed [19750.203583] do_IRQ: 0.41 No irq handler for vector [19750.215102] do_IRQ: 0.41 No irq handler for vector [19750.216939] do_IRQ: 0.41 No irq handler for vector [19750.222805] do_IRQ: 0.41 No irq handler for vector [19750.228827] do_IRQ: 0.41 No irq handler for vector [19750.235913] do_IRQ: 0.41 No irq handler for vector [19750.241491] do_IRQ: 0.41 No irq handler for vector [19750.252571] do_IRQ: 0.41 No irq handler for vector [19750.261505] do_IRQ: 0.41 No irq handler for vector [19750.263311] do_IRQ: 0.41 No irq handler for vector [19750.609506] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [19750.643478] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [19751.616544] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [19751.721351] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [19751.725357] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [19751.734817] Lustre: Skipped 1 previous similar message [19756.184859] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [19756.192709] do_IRQ: 7 callbacks suppressed [19756.192719] do_IRQ: 0.41 No irq handler for vector [19756.195328] do_IRQ: 0.41 No irq handler for vector [19756.197500] do_IRQ: 0.41 No irq handler for vector [19756.202619] do_IRQ: 0.41 No irq handler for vector [19756.207602] do_IRQ: 0.41 No irq handler for vector [19756.211144] do_IRQ: 0.41 No irq handler for vector [19756.215219] do_IRQ: 0.41 No irq handler for vector [19756.222956] do_IRQ: 0.41 No irq handler for vector [19756.226422] do_IRQ: 0.41 No irq handler for vector [19756.228759] do_IRQ: 0.41 No irq handler for vector [19759.964717] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 05:21:08 (1790760068) [19767.639725] Lustre: Failing over lustre-OST0000 [19767.642230] do_IRQ: 3 callbacks suppressed [19767.642241] do_IRQ: 0.41 No irq handler for vector [19767.644783] do_IRQ: 0.41 No irq handler for vector [19767.649679] do_IRQ: 0.41 No irq handler for vector [19767.652291] do_IRQ: 0.41 No irq handler for vector [19767.655145] do_IRQ: 0.41 No irq handler for vector [19767.657460] do_IRQ: 0.41 No irq handler for vector [19767.659836] do_IRQ: 0.41 No irq handler for vector [19767.661739] do_IRQ: 0.41 No irq handler for vector [19767.664646] do_IRQ: 0.41 No irq handler for vector [19767.667767] do_IRQ: 0.41 No irq handler for vector [19769.796465] Lustre: server umount lustre-OST0000 complete [19786.957458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19786.963925] do_IRQ: 3 callbacks suppressed [19786.963936] do_IRQ: 0.41 No irq handler for vector [19786.966361] do_IRQ: 0.41 No irq handler for vector [19786.972364] do_IRQ: 0.41 No irq handler for vector [19786.974993] do_IRQ: 0.41 No irq handler for vector [19786.976842] do_IRQ: 0.41 No irq handler for vector [19786.983226] do_IRQ: 0.41 No irq handler for vector [19786.984988] do_IRQ: 0.41 No irq handler for vector [19786.991565] do_IRQ: 0.41 No irq handler for vector [19786.994097] do_IRQ: 0.41 No irq handler for vector [19786.996973] do_IRQ: 0.41 No irq handler for vector [19787.347068] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [19787.366574] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [19787.454857] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [19789.145378] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [19789.146308] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [19789.156867] Lustre: Skipped 1 previous similar message [19793.027492] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [19793.034203] do_IRQ: 8 callbacks suppressed [19793.034212] do_IRQ: 0.41 No irq handler for vector [19793.037253] do_IRQ: 0.41 No irq handler for vector [19793.041974] do_IRQ: 0.41 No irq handler for vector [19793.045234] do_IRQ: 0.41 No irq handler for vector [19793.047071] do_IRQ: 0.41 No irq handler for vector [19793.050499] do_IRQ: 0.41 No irq handler for vector [19793.055156] do_IRQ: 0.41 No irq handler for vector [19793.059825] do_IRQ: 0.41 No irq handler for vector [19793.063102] do_IRQ: 0.41 No irq handler for vector [19793.066049] do_IRQ: 0.41 No irq handler for vector [19796.166138] Lustre: DEBUG MARKER: == sanity test 165g: ofd_access_log_reader --keepalive works ========================================================== 05:21:45 (1790760105) [19839.677374] Lustre: Failing over lustre-OST0000 [19839.682727] do_IRQ: 3 callbacks suppressed [19839.682736] do_IRQ: 0.41 No irq handler for vector [19839.688862] do_IRQ: 0.41 No irq handler for vector [19839.695218] do_IRQ: 0.41 No irq handler for vector [19839.699167] do_IRQ: 0.41 No irq handler for vector [19839.701529] do_IRQ: 0.41 No irq handler for vector [19839.703787] do_IRQ: 0.41 No irq handler for vector [19839.705940] do_IRQ: 0.41 No irq handler for vector [19839.708083] do_IRQ: 0.41 No irq handler for vector [19839.710073] do_IRQ: 0.41 No irq handler for vector [19839.712196] do_IRQ: 0.41 No irq handler for vector [19839.719270] LustreError: 264112:0:(ldlm_resource.c:1227:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000403:0xa058:0x0].0x0 (ffff8eb139cc5a00) refcount nonzero (2) after lock cleanup; forcing cleanup. [19839.813842] Lustre: server umount lustre-OST0000 complete [19840.481611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19840.485205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19840.494505] LustreError: Skipped 1 previous similar message [19840.516068] Lustre: Skipped 4 previous similar messages [19856.908291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19856.914770] do_IRQ: 8 callbacks suppressed [19856.914780] do_IRQ: 0.41 No irq handler for vector [19856.917495] do_IRQ: 0.41 No irq handler for vector [19856.921312] do_IRQ: 0.41 No irq handler for vector [19856.924794] do_IRQ: 0.41 No irq handler for vector [19856.928145] do_IRQ: 0.41 No irq handler for vector [19856.931574] do_IRQ: 0.41 No irq handler for vector [19856.934844] do_IRQ: 0.41 No irq handler for vector [19856.937720] do_IRQ: 0.41 No irq handler for vector [19856.940429] do_IRQ: 0.41 No irq handler for vector [19856.944202] do_IRQ: 0.41 No irq handler for vector [19857.230277] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [19857.255976] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [19858.451578] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [19858.699252] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [19858.710397] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [19858.718273] Lustre: Skipped 1 previous similar message [19863.090562] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing set_default_debug all all [19863.097031] do_IRQ: 8 callbacks suppressed [19863.097040] do_IRQ: 0.41 No irq handler for vector [19863.102347] do_IRQ: 0.41 No irq handler for vector [19863.107140] do_IRQ: 0.41 No irq handler for vector [19863.109943] do_IRQ: 0.41 No irq handler for vector [19863.113494] do_IRQ: 0.41 No irq handler for vector [19863.115925] do_IRQ: 0.41 No irq handler for vector [19863.118514] do_IRQ: 0.41 No irq handler for vector [19863.120657] do_IRQ: 0.41 No irq handler for vector [19863.123181] do_IRQ: 0.41 No irq handler for vector [19863.125373] do_IRQ: 0.41 No irq handler for vector [19866.325607] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 05:22:55 (1790760175) [19867.795843] Lustre: DEBUG MARKER: creating a 10 Mb file [19924.139285] Lustre: DEBUG MARKER: starting reads [19924.144873] do_IRQ: 3 callbacks suppressed [19924.144885] do_IRQ: 0.41 No irq handler for vector [19924.160085] do_IRQ: 0.41 No irq handler for vector [19924.176595] do_IRQ: 0.41 No irq handler for vector [19924.178651] do_IRQ: 0.41 No irq handler for vector [19924.184265] do_IRQ: 0.41 No irq handler for vector [19924.190163] do_IRQ: 0.41 No irq handler for vector [19924.195232] do_IRQ: 0.41 No irq handler for vector [19924.198761] do_IRQ: 0.41 No irq handler for vector [19924.204144] do_IRQ: 0.41 No irq handler for vector [19924.207463] do_IRQ: 0.41 No irq handler for vector [19925.763697] Lustre: DEBUG MARKER: truncating the file [19927.763038] Lustre: DEBUG MARKER: killing dd [19929.797839] Lustre: DEBUG MARKER: removing the temporary file [19929.804069] do_IRQ: 3 callbacks suppressed [19929.804076] do_IRQ: 0.41 No irq handler for vector [19929.808616] do_IRQ: 0.41 No irq handler for vector [19929.814490] do_IRQ: 0.41 No irq handler for vector [19929.817196] do_IRQ: 0.41 No irq handler for vector [19929.820410] do_IRQ: 0.41 No irq handler for vector [19929.823464] do_IRQ: 0.41 No irq handler for vector [19929.825980] do_IRQ: 0.41 No irq handler for vector [19929.831114] do_IRQ: 0.41 No irq handler for vector [19929.837438] do_IRQ: 0.41 No irq handler for vector [19929.841390] do_IRQ: 0.41 No irq handler for vector [19937.208917] Lustre: DEBUG MARKER: == sanity test 170a: test lctl df to handle corrupted log ========================================================== 05:24:06 (1790760246) [19937.217865] do_IRQ: 1 callbacks suppressed [19937.217876] do_IRQ: 0.41 No irq handler for vector [19937.225691] do_IRQ: 0.41 No irq handler for vector [19937.228848] do_IRQ: 0.41 No irq handler for vector [19937.236127] do_IRQ: 0.41 No irq handler for vector [19937.238518] do_IRQ: 0.41 No irq handler for vector [19937.244516] do_IRQ: 0.41 No irq handler for vector [19937.251262] do_IRQ: 0.41 No irq handler for vector [19937.256756] do_IRQ: 0.41 No irq handler for vector [19937.259861] do_IRQ: 0.41 No irq handler for vector [19937.265100] do_IRQ: 0.41 No irq handler for vector [19944.494551] Lustre: DEBUG MARKER: == sanity test 170b: check filename encoding ============= 05:24:13 (1790760253) [19944.505112] do_IRQ: 1 callbacks suppressed [19944.505120] do_IRQ: 0.41 No irq handler for vector [19944.510826] do_IRQ: 0.41 No irq handler for vector [19944.513768] do_IRQ: 0.41 No irq handler for vector [19944.516611] do_IRQ: 0.41 No irq handler for vector [19944.519223] do_IRQ: 0.41 No irq handler for vector [19944.522862] do_IRQ: 0.41 No irq handler for vector [19944.527748] do_IRQ: 0.41 No irq handler for vector [19944.529720] do_IRQ: 0.41 No irq handler for vector [19944.532709] do_IRQ: 0.41 No irq handler for vector [19944.534528] do_IRQ: 0.41 No irq handler for vector [19973.437181] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 05:24:42 (1790760282) [19973.455647] do_IRQ: 1 callbacks suppressed [19973.455658] do_IRQ: 0.41 No irq handler for vector [19973.470293] do_IRQ: 0.41 No irq handler for vector [19973.476642] do_IRQ: 0.41 No irq handler for vector [19973.485462] do_IRQ: 0.41 No irq handler for vector [19973.492321] do_IRQ: 0.41 No irq handler for vector [19973.497289] do_IRQ: 0.41 No irq handler for vector [19973.505543] do_IRQ: 0.41 No irq handler for vector [19973.522196] do_IRQ: 0.41 No irq handler for vector [19973.529422] do_IRQ: 0.41 No irq handler for vector [19973.536222] do_IRQ: 0.41 No irq handler for vector [19983.106242] Lustre: DEBUG MARKER: == sanity test 172: manual device removal with lctl cleanup/detach ================================================================ 05:24:52 (1790760292) [19983.114477] do_IRQ: 2 callbacks suppressed [19983.114487] do_IRQ: 0.41 No irq handler for vector [19983.122847] do_IRQ: 0.41 No irq handler for vector [19983.132207] do_IRQ: 0.41 No irq handler for vector [19983.135819] do_IRQ: 0.41 No irq handler for vector [19983.138335] do_IRQ: 0.41 No irq handler for vector [19983.143264] do_IRQ: 0.41 No irq handler for vector [19983.145899] do_IRQ: 0.41 No irq handler for vector [19983.151182] do_IRQ: 0.41 No irq handler for vector [19983.154188] do_IRQ: 0.41 No irq handler for vector [19983.159518] do_IRQ: 0.41 No irq handler for vector [19991.740034] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 05:25:00 (1790760300) [19991.747781] do_IRQ: 1 callbacks suppressed [19991.747794] do_IRQ: 0.41 No irq handler for vector [19991.754332] do_IRQ: 0.41 No irq handler for vector [19991.769296] do_IRQ: 0.41 No irq handler for vector [19991.774322] do_IRQ: 0.41 No irq handler for vector [19991.776772] do_IRQ: 0.41 No irq handler for vector [19991.779684] do_IRQ: 0.41 No irq handler for vector [19991.782289] do_IRQ: 0.41 No irq handler for vector [19991.785606] do_IRQ: 0.41 No irq handler for vector [19991.787995] do_IRQ: 0.41 No irq handler for vector [19991.794394] do_IRQ: 0.41 No irq handler for vector [19993.892989] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [19996.383996] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 05:25:04 (1790760304) [19999.933347] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_module obdecho/obdecho [19999.937912] do_IRQ: 3 callbacks suppressed [19999.937924] do_IRQ: 0.41 No irq handler for vector [19999.949961] do_IRQ: 0.41 No irq handler for vector [19999.956907] do_IRQ: 0.41 No irq handler for vector [19999.966836] do_IRQ: 0.41 No irq handler for vector [19999.970896] do_IRQ: 0.41 No irq handler for vector [19999.973880] do_IRQ: 0.41 No irq handler for vector [19999.976948] do_IRQ: 0.41 No irq handler for vector [19999.980447] do_IRQ: 0.41 No irq handler for vector [19999.985768] do_IRQ: 0.41 No irq handler for vector [19999.989221] do_IRQ: 0.41 No irq handler for vector [20017.223831] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 05:25:26 (1790760326) [20017.234032] do_IRQ: 1 callbacks suppressed [20017.234044] do_IRQ: 0.41 No irq handler for vector [20017.242643] do_IRQ: 0.41 No irq handler for vector [20017.245363] do_IRQ: 0.41 No irq handler for vector [20017.248278] do_IRQ: 0.41 No irq handler for vector [20017.249468] do_IRQ: 0.41 No irq handler for vector [20017.254055] do_IRQ: 0.41 No irq handler for vector [20017.257126] do_IRQ: 0.41 No irq handler for vector [20017.260918] do_IRQ: 0.41 No irq handler for vector [20017.274593] do_IRQ: 0.41 No irq handler for vector [20017.281333] do_IRQ: 0.41 No irq handler for vector [20022.025467] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing load_module obdecho/obdecho [20022.185703] Lustre: Echo OBD driver; http://www.lustre.org/ [20042.932058] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 05:25:51 (1790760351) [20042.950250] do_IRQ: 3 callbacks suppressed [20042.950261] do_IRQ: 0.41 No irq handler for vector [20042.961672] do_IRQ: 0.41 No irq handler for vector [20042.963036] do_IRQ: 0.41 No irq handler for vector [20042.968588] do_IRQ: 0.41 No irq handler for vector [20042.972693] do_IRQ: 0.41 No irq handler for vector [20042.979990] do_IRQ: 0.41 No irq handler for vector [20042.988100] do_IRQ: 0.41 No irq handler for vector [20042.998876] do_IRQ: 0.41 No irq handler for vector [20043.008972] do_IRQ: 0.41 No irq handler for vector [20043.017826] do_IRQ: 0.41 No irq handler for vector [20126.709713] Lustre: DEBUG MARKER: == sanity test 182a: Test parallel modify metadata operations from mdc ========================================================== 05:27:15 (1790760435) [20126.727046] do_IRQ: 1 callbacks suppressed [20126.727130] do_IRQ: 0.41 No irq handler for vector [20126.731032] do_IRQ: 0.41 No irq handler for vector [20126.743026] do_IRQ: 0.41 No irq handler for vector [20126.755085] do_IRQ: 0.41 No irq handler for vector [20126.759905] do_IRQ: 0.41 No irq handler for vector [20126.765226] do_IRQ: 0.41 No irq handler for vector [20126.769765] do_IRQ: 0.41 No irq handler for vector [20126.773678] do_IRQ: 0.41 No irq handler for vector [20126.780807] do_IRQ: 0.41 No irq handler for vector [20126.784725] do_IRQ: 0.41 No irq handler for vector [20210.465781] Lustre: DEBUG MARKER: == sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 05:28:39 (1790760519) [20210.477263] do_IRQ: 1 callbacks suppressed [20210.477275] do_IRQ: 0.41 No irq handler for vector [20210.491754] do_IRQ: 0.41 No irq handler for vector [20210.500442] do_IRQ: 0.41 No irq handler for vector [20210.505383] do_IRQ: 0.41 No irq handler for vector [20210.510472] do_IRQ: 0.41 No irq handler for vector [20210.513257] do_IRQ: 0.41 No irq handler for vector [20210.514897] do_IRQ: 0.41 No irq handler for vector [20210.517568] do_IRQ: 0.41 No irq handler for vector [20210.519728] do_IRQ: 0.41 No irq handler for vector [20210.523949] do_IRQ: 0.41 No irq handler for vector [20901.221354] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 05:40:10 (1790761210) [20901.230017] do_IRQ: 1 callbacks suppressed [20901.230025] do_IRQ: 0.41 No irq handler for vector [20901.236592] do_IRQ: 0.41 No irq handler for vector [20901.241146] do_IRQ: 0.41 No irq handler for vector [20901.243391] do_IRQ: 0.41 No irq handler for vector [20901.246710] do_IRQ: 0.41 No irq handler for vector [20901.253360] do_IRQ: 0.41 No irq handler for vector [20901.255149] do_IRQ: 0.41 No irq handler for vector [20901.265305] do_IRQ: 0.41 No irq handler for vector [20901.268780] do_IRQ: 0.41 No irq handler for vector [20901.273245] do_IRQ: 0.41 No irq handler for vector [20902.689046] Lustre: *** cfs_fail_loc=148, val=0*** [20911.454159] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 05:40:20 (1790761220) [20911.459365] do_IRQ: 2 callbacks suppressed [20911.459374] do_IRQ: 0.41 No irq handler for vector [20911.461717] do_IRQ: 0.41 No irq handler for vector [20911.463784] do_IRQ: 0.41 No irq handler for vector [20911.466058] do_IRQ: 0.41 No irq handler for vector [20911.467452] do_IRQ: 0.41 No irq handler for vector [20911.468950] do_IRQ: 0.41 No irq handler for vector [20911.470303] do_IRQ: 0.41 No irq handler for vector [20911.472373] do_IRQ: 0.41 No irq handler for vector [20911.474549] do_IRQ: 0.41 No irq handler for vector [20911.476243] do_IRQ: 0.41 No irq handler for vector [20924.177439] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 05:40:32 (1790761232) [20924.187742] do_IRQ: 2 callbacks suppressed [20924.187759] do_IRQ: 0.41 No irq handler for vector [20924.197146] do_IRQ: 0.41 No irq handler for vector [20924.206456] do_IRQ: 0.41 No irq handler for vector [20924.218356] do_IRQ: 0.41 No irq handler for vector [20924.222573] do_IRQ: 0.41 No irq handler for vector [20924.224837] do_IRQ: 0.41 No irq handler for vector [20924.236731] do_IRQ: 0.41 No irq handler for vector [20924.241987] do_IRQ: 0.41 No irq handler for vector [20924.244670] do_IRQ: 0.41 No irq handler for vector [20924.250920] do_IRQ: 0.41 No irq handler for vector [20934.175577] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 05:40:42 (1790761242) [20934.186246] do_IRQ: 1 callbacks suppressed [20934.186254] do_IRQ: 0.41 No irq handler for vector [20934.198263] do_IRQ: 0.41 No irq handler for vector [20934.206549] do_IRQ: 0.41 No irq handler for vector [20934.210955] do_IRQ: 0.41 No irq handler for vector [20934.218457] do_IRQ: 0.41 No irq handler for vector [20934.224664] do_IRQ: 0.41 No irq handler for vector [20934.233570] do_IRQ: 0.41 No irq handler for vector [20934.247162] do_IRQ: 0.41 No irq handler for vector [20934.254135] do_IRQ: 0.41 No irq handler for vector [20934.266024] do_IRQ: 0.41 No irq handler for vector [20986.361721] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 05:41:35 (1790761295) [20986.365835] do_IRQ: 1 callbacks suppressed [20986.365846] do_IRQ: 0.41 No irq handler for vector [20986.374058] do_IRQ: 0.41 No irq handler for vector [20986.376931] do_IRQ: 0.41 No irq handler for vector [20986.382504] do_IRQ: 0.41 No irq handler for vector [20986.384653] do_IRQ: 0.41 No irq handler for vector [20986.386284] do_IRQ: 0.41 No irq handler for vector [20986.387816] do_IRQ: 0.41 No irq handler for vector [20986.389560] do_IRQ: 0.41 No irq handler for vector [20986.394602] do_IRQ: 0.41 No irq handler for vector [20986.398980] do_IRQ: 0.41 No irq handler for vector [21008.137292] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 05:41:57 (1790761317) [21008.156641] do_IRQ: 1 callbacks suppressed [21008.156648] do_IRQ: 0.41 No irq handler for vector [21008.167796] do_IRQ: 0.41 No irq handler for vector [21008.175468] do_IRQ: 0.41 No irq handler for vector [21008.184456] do_IRQ: 0.41 No irq handler for vector [21008.190976] do_IRQ: 0.41 No irq handler for vector [21008.195645] do_IRQ: 0.41 No irq handler for vector [21008.207644] do_IRQ: 0.41 No irq handler for vector [21008.213258] do_IRQ: 0.41 No irq handler for vector [21008.219561] do_IRQ: 0.41 No irq handler for vector [21008.221141] do_IRQ: 0.41 No irq handler for vector [21029.049685] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 05:42:17 (1790761337) [21029.057395] do_IRQ: 1 callbacks suppressed [21029.057404] do_IRQ: 0.41 No irq handler for vector [21029.059899] do_IRQ: 0.41 No irq handler for vector [21029.061566] do_IRQ: 0.41 No irq handler for vector [21029.064555] do_IRQ: 0.41 No irq handler for vector [21029.067148] do_IRQ: 0.41 No irq handler for vector [21029.068883] do_IRQ: 0.41 No irq handler for vector [21029.071633] do_IRQ: 0.41 No irq handler for vector [21029.076995] do_IRQ: 0.41 No irq handler for vector [21029.081497] do_IRQ: 0.41 No irq handler for vector [21029.086849] do_IRQ: 0.41 No irq handler for vector [21036.917030] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 05:42:26 (1790761346) [21036.921304] do_IRQ: 2 callbacks suppressed [21036.921317] do_IRQ: 0.41 No irq handler for vector [21036.943336] do_IRQ: 0.41 No irq handler for vector [21036.947549] do_IRQ: 0.41 No irq handler for vector [21036.950089] do_IRQ: 0.41 No irq handler for vector [21036.953929] do_IRQ: 0.41 No irq handler for vector [21036.960050] do_IRQ: 0.41 No irq handler for vector [21036.961852] do_IRQ: 0.41 No irq handler for vector [21036.966295] do_IRQ: 0.41 No irq handler for vector [21036.969919] do_IRQ: 0.41 No irq handler for vector [21036.973431] do_IRQ: 0.41 No irq handler for vector [21045.991367] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 05:42:35 (1790761355) [21045.999955] do_IRQ: 1 callbacks suppressed [21045.999965] do_IRQ: 0.41 No irq handler for vector [21046.009960] do_IRQ: 0.41 No irq handler for vector [21046.019629] do_IRQ: 0.41 No irq handler for vector [21046.021842] do_IRQ: 0.41 No irq handler for vector [21046.024213] do_IRQ: 0.41 No irq handler for vector [21046.028801] do_IRQ: 0.41 No irq handler for vector [21046.030476] do_IRQ: 0.41 No irq handler for vector [21046.036394] do_IRQ: 0.41 No irq handler for vector [21046.040996] do_IRQ: 0.41 No irq handler for vector [21046.046405] do_IRQ: 0.41 No irq handler for vector [21058.391101] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 05:42:47 (1790761367) [21058.408878] do_IRQ: 1 callbacks suppressed [21058.408885] do_IRQ: 0.41 No irq handler for vector [21058.420509] do_IRQ: 0.41 No irq handler for vector [21058.429197] do_IRQ: 0.41 No irq handler for vector [21058.434414] do_IRQ: 0.41 No irq handler for vector [21058.443360] do_IRQ: 0.41 No irq handler for vector [21058.446149] do_IRQ: 0.41 No irq handler for vector [21058.450263] do_IRQ: 0.41 No irq handler for vector [21058.455699] do_IRQ: 0.41 No irq handler for vector [21058.458269] do_IRQ: 0.41 No irq handler for vector [21058.463509] do_IRQ: 0.41 No irq handler for vector [21067.242940] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 05:42:56 (1790761376) [21067.279063] do_IRQ: 1 callbacks suppressed [21067.279073] do_IRQ: 0.41 No irq handler for vector [21067.296360] do_IRQ: 0.41 No irq handler for vector [21067.305801] do_IRQ: 0.41 No irq handler for vector [21067.312899] do_IRQ: 0.41 No irq handler for vector [21067.322672] do_IRQ: 0.41 No irq handler for vector [21067.329065] do_IRQ: 0.41 No irq handler for vector [21067.338812] do_IRQ: 0.41 No irq handler for vector [21067.341948] do_IRQ: 0.41 No irq handler for vector [21067.351331] do_IRQ: 0.41 No irq handler for vector [21067.355599] do_IRQ: 0.41 No irq handler for vector [21075.212421] Lustre: DEBUG MARKER: == sanity test 190a: check lfs project -p works with project name ========================================================== 05:43:03 (1790761383) [21075.230390] do_IRQ: 1 callbacks suppressed [21075.230413] do_IRQ: 0.41 No irq handler for vector [21075.248470] do_IRQ: 0.41 No irq handler for vector [21075.272143] do_IRQ: 0.41 No irq handler for vector [21075.277834] do_IRQ: 0.41 No irq handler for vector [21075.285614] do_IRQ: 0.41 No irq handler for vector [21075.287379] do_IRQ: 0.41 No irq handler for vector [21075.302971] do_IRQ: 0.41 No irq handler for vector [21075.310930] do_IRQ: 0.41 No irq handler for vector [21075.317923] do_IRQ: 0.41 No irq handler for vector [21075.323617] do_IRQ: 0.41 No irq handler for vector [21084.283607] Lustre: DEBUG MARKER: == sanity test 190b: check lfs find --project works with project name ========================================================== 05:43:13 (1790761393) [21084.306638] do_IRQ: 1 callbacks suppressed [21084.306645] do_IRQ: 0.41 No irq handler for vector [21084.314471] do_IRQ: 0.41 No irq handler for vector [21084.317298] do_IRQ: 0.41 No irq handler for vector [21084.319957] do_IRQ: 0.41 No irq handler for vector [21084.330315] do_IRQ: 0.41 No irq handler for vector [21084.334996] do_IRQ: 0.41 No irq handler for vector [21084.353594] do_IRQ: 0.41 No irq handler for vector [21084.357686] do_IRQ: 0.41 No irq handler for vector [21084.367009] do_IRQ: 0.41 No irq handler for vector [21084.368787] do_IRQ: 0.41 No irq handler for vector [21175.499959] Lustre: DEBUG MARKER: == sanity test 190c: check lfs project -p works with u:USERNAME ========================================================== 05:44:44 (1790761484) [21175.521547] do_IRQ: 1 callbacks suppressed [21175.521559] do_IRQ: 0.41 No irq handler for vector [21175.526026] do_IRQ: 0.41 No irq handler for vector [21175.533231] do_IRQ: 0.41 No irq handler for vector [21175.538586] do_IRQ: 0.41 No irq handler for vector [21175.542692] do_IRQ: 0.41 No irq handler for vector [21175.548375] do_IRQ: 0.41 No irq handler for vector [21175.551828] do_IRQ: 0.41 No irq handler for vector [21175.555623] do_IRQ: 0.41 No irq handler for vector [21175.558822] do_IRQ: 0.41 No irq handler for vector [21175.565949] do_IRQ: 0.41 No irq handler for vector [21194.513309] Lustre: DEBUG MARKER: == sanity test complete, duration 20936 sec ============== 05:45:03 (1790761503) [21194.518838] do_IRQ: 2 callbacks suppressed [21194.518851] do_IRQ: 0.41 No irq handler for vector [21194.521580] do_IRQ: 0.41 No irq handler for vector [21194.527273] do_IRQ: 0.41 No irq handler for vector [21194.535880] do_IRQ: 0.41 No irq handler for vector [21194.540299] do_IRQ: 0.41 No irq handler for vector [21194.543840] do_IRQ: 0.41 No irq handler for vector [21194.545834] do_IRQ: 0.41 No irq handler for vector [21194.547890] do_IRQ: 0.41 No irq handler for vector [21194.550807] do_IRQ: 0.41 No irq handler for vector [21194.552884] do_IRQ: 0.41 No irq handler for vector [21196.296273] Lustre: DEBUG MARKER: === sanity: start cleanup 05:45:05 (1790761505) === [21250.643138] Lustre: DEBUG MARKER: === sanity: finish cleanup 05:45:59 (1790761559) === [21250.649089] do_IRQ: 2 callbacks suppressed [21250.649096] do_IRQ: 0.41 No irq handler for vector [21250.657845] do_IRQ: 0.41 No irq handler for vector [21250.664894] do_IRQ: 0.41 No irq handler for vector [21250.674983] do_IRQ: 0.41 No irq handler for vector [21250.681304] do_IRQ: 0.41 No irq handler for vector [21250.686190] do_IRQ: 0.41 No irq handler for vector [21250.688815] do_IRQ: 0.41 No irq handler for vector [21250.693737] do_IRQ: 0.41 No irq handler for vector [21250.698725] do_IRQ: 0.41 No irq handler for vector [21250.708911] do_IRQ: 0.41 No irq handler for vector [21256.285860] Lustre: server umount lustre-MDT0000 complete [21256.289332] do_IRQ: 1 callbacks suppressed [21256.289355] do_IRQ: 0.41 No irq handler for vector [21256.316623] do_IRQ: 0.41 No irq handler for vector [21256.321554] do_IRQ: 0.41 No irq handler for vector [21256.327334] do_IRQ: 0.41 No irq handler for vector [21256.337272] do_IRQ: 0.41 No irq handler for vector [21256.343769] do_IRQ: 0.41 No irq handler for vector [21256.347640] do_IRQ: 0.41 No irq handler for vector [21256.350402] do_IRQ: 0.41 No irq handler for vector [21256.352892] do_IRQ: 0.41 No irq handler for vector [21256.355575] do_IRQ: 0.41 No irq handler for vector [21257.704235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [21257.724150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [21257.754923] LustreError: 123689:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [21257.773758] LustreError: 123689:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [21264.143690] LustreError: 188426:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790761575 with bad export cookie 17064066237235786182 [21264.149799] LustreError: MGC192.168.206.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [21264.166017] do_IRQ: 5 callbacks suppressed [21264.166027] do_IRQ: 0.41 No irq handler for vector [21264.166039] LustreError: 188426:0:(ldlm_lockd.c:2567:ldlm_cancel_handler()) Skipped 1 previous similar message [21264.180871] do_IRQ: 0.41 No irq handler for vector [21264.185218] do_IRQ: 0.41 No irq handler for vector [21264.217150] do_IRQ: 0.41 No irq handler for vector [21264.223451] do_IRQ: 0.41 No irq handler for vector [21264.227498] do_IRQ: 0.41 No irq handler for vector [21264.232261] do_IRQ: 0.41 No irq handler for vector [21264.242305] do_IRQ: 0.41 No irq handler for vector [21264.245120] do_IRQ: 0.41 No irq handler for vector [21264.246793] do_IRQ: 0.41 No irq handler for vector [21264.611804] Lustre: server umount lustre-MDT0001 complete [21281.761264] Lustre: 120260:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790761576/real 1790761576] req@ffff8eb1324f8380 x1877740495419520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790761592 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [21281.795804] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [21281.799334] do_IRQ: 2 callbacks suppressed [21281.799346] do_IRQ: 0.41 No irq handler for vector [21281.814012] Lustre: Skipped 3 previous similar messages [21281.818088] do_IRQ: 0.41 No irq handler for vector [21281.820700] do_IRQ: 0.41 No irq handler for vector [21281.828920] do_IRQ: 0.41 No irq handler for vector [21281.831294] do_IRQ: 0.41 No irq handler for vector [21281.833676] do_IRQ: 0.41 No irq handler for vector [21281.836273] do_IRQ: 0.41 No irq handler for vector [21281.838828] do_IRQ: 0.41 No irq handler for vector [21281.841636] do_IRQ: 0.41 No irq handler for vector [21281.843643] do_IRQ: 0.41 No irq handler for vector [21284.079734] Lustre: server umount lustre-OST0000 complete [21285.344418] Lustre: 120257:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790761580/real 1790761580] req@ffff8eb1332e3b80 x1877740495419904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790761596 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [21286.944159] Lustre: 120260:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790761581/real 1790761581] req@ffff8eb138ecd500 x1877740495420160/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790761597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [21286.975217] do_IRQ: 3 callbacks suppressed [21286.975227] do_IRQ: 0.41 No irq handler for vector [21286.979853] do_IRQ: 0.41 No irq handler for vector [21286.984808] do_IRQ: 0.41 No irq handler for vector [21286.986537] do_IRQ: 0.41 No irq handler for vector [21286.991213] do_IRQ: 0.41 No irq handler for vector [21287.001488] do_IRQ: 0.41 No irq handler for vector [21287.005919] do_IRQ: 0.41 No irq handler for vector [21287.008342] do_IRQ: 0.41 No irq handler for vector [21287.010398] do_IRQ: 0.41 No irq handler for vector [21287.012194] do_IRQ: 0.41 No irq handler for vector [21290.532113] Lustre: 120257:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790761585/real 1790761585] req@ffff8eb12fc8e680 x1877740495420672/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790761601 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [21292.428902] Lustre: server umount lustre-OST0001 complete [21292.441530] do_IRQ: 2 callbacks suppressed [21292.441537] do_IRQ: 0.41 No irq handler for vector [21292.462920] do_IRQ: 0.41 No irq handler for vector [21292.468153] do_IRQ: 0.41 No irq handler for vector [21292.476457] do_IRQ: 0.41 No irq handler for vector [21292.479280] do_IRQ: 0.41 No irq handler for vector [21292.490779] do_IRQ: 0.41 No irq handler for vector [21292.496891] do_IRQ: 0.41 No irq handler for vector [21292.503356] do_IRQ: 0.41 No irq handler for vector [21292.508595] do_IRQ: 0.41 No irq handler for vector [21292.513388] do_IRQ: 0.41 No irq handler for vector [21308.079343] Lustre: DEBUG MARKER: oleg629-server.virtnet: executing unload_modules_local [21308.083976] do_IRQ: 1 callbacks suppressed [21308.083986] do_IRQ: 0.41 No irq handler for vector [21308.102076] do_IRQ: 0.41 No irq handler for vector [21308.109422] do_IRQ: 0.41 No irq handler for vector [21308.111049] do_IRQ: 0.41 No irq handler for vector [21308.117964] do_IRQ: 0.41 No irq handler for vector [21308.121476] do_IRQ: 0.41 No irq handler for vector [21308.125151] do_IRQ: 0.41 No irq handler for vector [21308.127459] do_IRQ: 0.41 No irq handler for vector [21308.130419] do_IRQ: 0.41 No irq handler for vector [21308.133497] do_IRQ: 0.41 No irq handler for vector [21310.874720] Key type lgssc unregistered [21311.127504] LNet: 285412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [21311.132771] LNetError: 285412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [21311.142284] LNet: Removed LNI 192.168.206.129@tcp [21311.917258] Key type .llcrypt unregistered [21311.927325] Key type ._llcrypt unregistered