[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 302532471 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.447332] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.451295] pid_max: default: 32768 minimum: 301 [ 0.454929] Security Framework initialized [ 0.456290] SELinux: Initializing. [ 0.457611] SELinux: Starting in permissive mode [ 0.459185] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.464062] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.468630] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.470438] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.473248] Initializing cgroup subsys memory [ 0.474796] Initializing cgroup subsys devices [ 0.476171] Initializing cgroup subsys freezer [ 0.477367] Initializing cgroup subsys net_cls [ 0.478925] Initializing cgroup subsys blkio [ 0.480096] Initializing cgroup subsys perf_event [ 0.481603] Initializing cgroup subsys hugetlb [ 0.483334] Initializing cgroup subsys pids [ 0.485675] Initializing cgroup subsys net_prio [ 0.487302] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.490778] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.492702] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.494225] tlb_flushall_shift: 6 [ 0.495268] FEATURE SPEC_CTRL Present [ 0.496232] FEATURE IBPB_SUPPORT Present [ 0.497784] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.500010] Spectre V2 : Vulnerable [ 0.501215] Speculative Store Bypass: Vulnerable [ 0.503850] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.512552] ACPI: Core revision 20130517 [ 0.516946] ACPI: All ACPI Tables successfully acquired [ 0.519031] ftrace: allocating 30294 entries in 119 pages [ 0.576615] Enabling x2apic [ 0.578190] Enabled x2apic [ 0.579441] Switched APIC routing to physical x2apic. [ 0.584439] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.587141] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.591106] TSC deadline timer enabled [ 0.591341] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.594214] ... version: 2 [ 0.595634] ... bit width: 48 [ 0.596691] ... generic registers: 4 [ 0.597759] ... value mask: 0000ffffffffffff [ 0.599271] ... max period: 00007fffffffffff [ 0.600843] ... fixed-purpose events: 3 [ 0.602262] ... event mask: 000000070000000f [ 0.604219] KVM setup paravirtual spinlock [ 0.610575] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.613481] KVM setup async PF for cpu 1 [ 0.615039] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.618004] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.621117] KVM setup async PF for cpu 2 [ 0.608398] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.621690] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.623917] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.625441] Brought up 4 CPUs [ 0.625454] KVM setup async PF for cpu 3 [ 0.625461] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.628517] smpboot: Max logical packages: 1 [ 0.629471] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.634581] devtmpfs: initialized [ 0.635999] x86/mm: Memory block size: 128MB [ 0.640100] EVM: security.selinux [ 0.641264] EVM: security.ima [ 0.642367] EVM: security.capability [ 0.646390] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.648724] NET: Registered protocol family 16 [ 0.650775] cpuidle: using governor haltpoll [ 0.652749] ACPI: bus type PCI registered [ 0.654049] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.656598] PCI: Using configuration type 1 for base access [ 0.658162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.666537] ACPI: Added _OSI(Module Device) [ 0.667642] ACPI: Added _OSI(Processor Device) [ 0.669268] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.670946] ACPI: Added _OSI(Processor Aggregator Device) [ 0.673011] ACPI: Added _OSI(Linux-Dell-Video) [ 0.675202] ACPI: EC: Look up EC in DSDT [ 0.677518] ACPI: Interpreter enabled [ 0.678796] ACPI: (supports S0 S3 S4 S5) [ 0.679977] ACPI: Using IOAPIC for interrupt routing [ 0.681680] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.685094] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.692331] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.693983] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.695901] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.698176] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.702715] acpiphp: Slot [2] registered [ 0.704231] acpiphp: Slot [3] registered [ 0.705652] acpiphp: Slot [4] registered [ 0.707147] acpiphp: Slot [5] registered [ 0.708571] acpiphp: Slot [6] registered [ 0.710000] acpiphp: Slot [7] registered [ 0.711560] acpiphp: Slot [8] registered [ 0.712915] acpiphp: Slot [9] registered [ 0.714362] acpiphp: Slot [10] registered [ 0.715976] acpiphp: Slot [11] registered [ 0.717574] acpiphp: Slot [12] registered [ 0.718866] acpiphp: Slot [13] registered [ 0.720130] acpiphp: Slot [14] registered [ 0.721712] acpiphp: Slot [15] registered [ 0.722569] acpiphp: Slot [16] registered [ 0.723997] acpiphp: Slot [17] registered [ 0.725166] acpiphp: Slot [18] registered [ 0.726628] acpiphp: Slot [19] registered [ 0.728242] acpiphp: Slot [20] registered [ 0.729681] acpiphp: Slot [21] registered [ 0.731131] acpiphp: Slot [22] registered [ 0.732486] acpiphp: Slot [23] registered [ 0.734075] acpiphp: Slot [24] registered [ 0.735395] acpiphp: Slot [25] registered [ 0.736481] acpiphp: Slot [26] registered [ 0.737620] acpiphp: Slot [27] registered [ 0.738428] acpiphp: Slot [28] registered [ 0.739867] acpiphp: Slot [29] registered [ 0.741348] acpiphp: Slot [30] registered [ 0.742563] acpiphp: Slot [31] registered [ 0.743527] PCI host bridge to bus 0000:00 [ 0.744821] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.746674] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.748614] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.751282] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.754003] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.756297] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.757973] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.758925] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.759963] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.765569] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.768735] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.771382] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.773672] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.776034] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.778545] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.779202] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.781368] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.783882] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.785814] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.798098] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.800205] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.802818] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.807125] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.811172] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.827372] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.835287] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.841808] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.846509] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.855620] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.862098] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.865003] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.867660] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.877798] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.883228] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.886422] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.889902] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.903059] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.906790] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.909647] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.915636] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.923028] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.930815] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.933489] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.935693] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.945392] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.949243] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.951355] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.956162] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.958007] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.959640] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.962938] vgaarb: loaded [ 0.963899] SCSI subsystem initialized [ 0.964925] ACPI: bus type USB registered [ 0.965933] usbcore: registered new interface driver usbfs [ 0.967316] usbcore: registered new interface driver hub [ 0.968851] usbcore: registered new device driver usb [ 0.970568] PCI: Using ACPI for IRQ routing [ 0.971736] PCI: pci_cache_line_size set to 64 bytes [ 0.972097] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.972104] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 0.972105] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 0.973152] NetLabel: Initializing [ 0.974191] NetLabel: domain hash size = 128 [ 0.975369] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.976664] NetLabel: unlabeled traffic allowed by default [ 0.978523] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.979885] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.988969] amd_nb: Cannot enumerate AMD northbridges [ 0.990518] Switched to clocksource kvm-clock [ 1.009342] pnp: PnP ACPI init [ 1.010275] ACPI: bus type PNP registered [ 1.011647] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.011748] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.011797] pnp 00:02: [dma 2] [ 1.011831] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.011904] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.011970] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.012034] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.012363] pnp: PnP ACPI: found 6 devices [ 1.013557] ACPI: bus type PNP unregistered [ 1.024969] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.024976] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.024979] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.024981] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.024982] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.025065] NET: Registered protocol family 2 [ 1.027073] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.029430] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.032241] TCP: Hash tables configured (established 32768 bind 32768) [ 1.034080] TCP: reno registered [ 1.035368] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.037494] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.039844] NET: Registered protocol family 1 [ 1.041806] RPC: Registered named UNIX socket transport module. [ 1.043963] RPC: Registered udp transport module. [ 1.045440] RPC: Registered tcp transport module. [ 1.046965] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.049157] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.051034] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.052836] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.054876] PCI: CLS 0 bytes, default 64 [ 1.055958] Unpacking initramfs... [ 2.407301] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.411134] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.413322] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.416662] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.419325] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.421154] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.423180] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.427108] cryptomgr_test (52) used greatest stack depth: 14480 bytes left [ 2.427680] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.427722] Initialise system trusted keyring [ 2.458925] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.461071] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.468235] zpool: loaded [ 2.469072] zbud: loaded [ 2.470457] VFS: Disk quotas dquot_6.6.0 [ 2.472073] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.475461] NFS: Registering the id_resolver key type [ 2.477262] Key type id_resolver registered [ 2.478585] Key type id_legacy registered [ 2.479803] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.482842] Key type big_key registered [ 2.484168] SELinux: Registering netfilter hooks [ 2.489611] cryptomgr_test (58) used greatest stack depth: 13968 bytes left [ 2.494731] cryptomgr_test (61) used greatest stack depth: 13664 bytes left [ 2.495354] NET: Registered protocol family 38 [ 2.495367] Key type asymmetric registered [ 2.495370] Asymmetric key parser 'x509' registered [ 2.495541] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.495654] io scheduler noop registered [ 2.495658] io scheduler deadline registered (default) [ 2.495722] io scheduler cfq registered [ 2.495727] io scheduler mq-deadline registered [ 2.495732] io scheduler kyber registered [ 2.498732] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.498742] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.517720] intel_idle: does not run on family 6 model 62 [ 2.517869] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.520085] ACPI: Power Button [PWRF] [ 2.521827] GHES: HEST is not enabled! [ 2.580480] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.641756] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.765364] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.831967] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 2.963995] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.997464] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 3.032573] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 3.037021] Non-volatile memory driver v1.3 [ 3.038948] Linux agpgart interface v0.103 [ 3.040987] crash memory driver: version 1.1 [ 3.043221] nbd: registered device at major 43 [ 3.048895] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 3.048948] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 3.049003] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 3.049043] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 3.049083] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 3.056648] virtio_blk virtio1: [vda] 115448 512-byte logical blocks (59.1 MB/56.3 MiB) [ 3.064238] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 3.064269] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 3.064299] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 3.064323] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 3.064348] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 3.070752] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.077094] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 3.077133] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 3.077173] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 3.077207] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 3.077239] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 3.083815] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.091741] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 3.091800] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 3.091838] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 3.091883] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 3.091924] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 3.099332] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.106453] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 3.106547] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 3.106594] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 3.106636] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 3.106678] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 3.113502] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.119970] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 3.120017] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 3.120058] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 3.120096] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 3.120143] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 3.126770] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.136284] rdac: device handler registered [ 3.138050] hp_sw: device handler registered [ 3.139565] emc: device handler registered [ 3.141037] libphy: Fixed MDIO Bus: probed [ 3.144998] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.147272] ehci-pci: EHCI PCI platform driver [ 3.148621] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.150542] ohci-pci: OHCI PCI platform driver [ 3.152311] uhci_hcd: USB Universal Host Controller Interface driver [ 3.154798] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.158601] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.159984] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.162080] mousedev: PS/2 mouse device common for all mice [ 3.164977] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.165262] rtc_cmos 00:05: RTC can wake from S4 [ 3.166492] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.167597] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.172412] hidraw: raw HID events driver (C) Jiri Kosina [ 3.172793] usbcore: registered new interface driver usbhid [ 3.172794] usbhid: USB HID core driver [ 3.172904] drop_monitor: Initializing network drop monitor service [ 3.172987] Netfilter messages via NETLINK v0.30. [ 3.173093] TCP: cubic registered [ 3.173100] Initializing XFRM netlink socket [ 3.173486] NET: Registered protocol family 10 [ 3.174139] NET: Registered protocol family 17 [ 3.174196] Key type dns_resolver registered [ 3.174737] mce: Using 10 MCE banks [ 3.175059] Loading compiled-in X.509 certificates [ 3.176474] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.176562] registered taskstats version 1 [ 3.179578] modprobe (71) used greatest stack depth: 13456 bytes left [ 3.183328] Key type trusted registered [ 3.188236] Key type encrypted registered [ 3.188306] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.191501] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.192710] rtc_cmos 00:05: setting system clock to 2025-05-02 00:53:27 UTC (1746147207) [ 3.218621] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.220876] Write protecting the kernel read-only data: 12288k [ 3.222326] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.225187] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.234162] random: systemd: uninitialized urandom read (16 bytes read) [ 3.237826] random: systemd: uninitialized urandom read (16 bytes read) [ 3.239788] random: systemd: uninitialized urandom read (16 bytes read) [ 3.243510] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.249670] systemd[1]: Detected virtualization kvm. [ 3.251342] systemd[1]: Detected architecture x86-64. [ 3.253167] systemd[1]: Running in initial RAM disk. [ 3.258209] systemd[1]: No hostname configured. [ 3.260024] systemd[1]: Set hostname to . [ 3.262020] random: systemd: uninitialized urandom read (16 bytes read) [ 3.264410] systemd[1]: Initializing machine ID from random generator. [ 3.276853] ln (88) used greatest stack depth: 13008 bytes left [ 3.319903] random: systemd: uninitialized urandom read (16 bytes read) [ 3.322569] random: systemd: uninitialized urandom read (16 bytes read) [ 3.325452] random: systemd: uninitialized urandom read (16 bytes read) [ 3.328052] random: systemd: uninitialized urandom read (16 bytes read) [ 3.331441] random: systemd: uninitialized urandom read (16 bytes read) [ 3.334033] random: systemd: uninitialized urandom read (16 bytes read) [ 3.344273] systemd[1]: Reached target Swap. [ 3.348620] systemd[1]: Created slice Root Slice. [ 3.353237] systemd[1]: Created slice System Slice. [ 3.357935] systemd[1]: Listening on Journal Socket. [ 3.364260] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.373706] systemd[1]: Starting Load Kernel Modules... [ 3.379045] systemd[1]: Starting dracut cmdline hook... [ 3.382758] systemd[1]: Listening on udev Kernel Socket. [ 3.388822] systemd[1]: Starting Setup Virtual Console... [ 3.392165] systemd[1]: Listening on udev Control Socket. [ 3.397661] systemd[1]: Reached target Sockets. [ 3.401036] systemd[1]: Reached target Local File Systems. [ 3.407648] systemd[1]: Starting Journal Service... [ 3.410867] systemd[1]: Reached target Timers. [ 3.414273] systemd[1]: Reached target Slices. [ 3.419613] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 3.426595] tsc: Refined TSC clocksource calibration: 2399.979 MHz [ 3.431778] systemd[1]: Started Load Kernel Modules. [ 3.436777] systemd[1]: Started Setup Virtual Console. [ 3.443007] systemd[1]: Starting Apply Kernel Variables... [ 3.448699] systemd[1]: Starting Create Static Device Nodes in /dev... [ 3.455826] systemd[1]: Started Journal Service. [ 3.467035] random: fast init done [ 3.974088] libata version 3.00 loaded. [ 3.976475] ata_piix 0000:00:01.1: version 2.13 [ 3.983299] scsi host0: ata_piix [ 3.990346] scsi host1: ata_piix [ 3.992537] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.996104] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 4.038479] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 4.246644] ip (343) used greatest stack depth: 12336 bytes left [ 7.392639] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 7.827048] systemd-journald[114]: Received SIGTERM from PID 1 (systemd). [ 8.077166] SELinux: Disabled at runtime. [ 8.079064] SELinux: Unregistering netfilter hooks [ 8.160988] ip_tables: (C) 2000-2006 Netfilter Core Team [ 8.164782] systemd[1]: Inserted module 'ip_tables' [ 8.627570] systemd-journald[567]: Received request to flush runtime journal from PID 1 [ 8.760959] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 8.774802] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 8.818284] cryptd: max_cpu_qlen set to 1000 [ 8.856440] AVX version of gcm_enc/dec engaged. [ 8.860036] AES CTR mode by8 optimization enabled [ 8.871184] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 8.932605] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 8.936308] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 8.939682] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 9.037939] EDAC MC: Ver: 3.0.0 [ 9.045869] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 9.045886] EDAC sbridge: Ver: 1.1.2 [ 11.308779] mount.nfs (772) used greatest stack depth: 10704 bytes left [ 22.218578] device-mapper: uevent: version 1.0.3 [ 22.220887] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 27.268553] libcfs: loading out-of-tree module taints kernel. [ 27.271094] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 27.304301] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 32.433580] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 32.657150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 32.662467] alg: No test for adler32 (adler32-zlib) [ 33.576418] Lustre: Lustre: Build Version: 2.16.54_69_g06f2fd3 [ 33.776483] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 33.778945] LNet: Accept secure, port 988 [ 35.331745] Key type lgssc registered [ 35.698779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 38.962469] icp: module license 'CDDL' taints kernel. [ 38.964390] Disabling lock debugging due to kernel taint [ 40.565620] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 43.938473] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 51.712720] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 56.228749] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 56.241516] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 60.409485] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 60.420538] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 66.899698] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 72.983368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 73.036445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 73.063286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 74.236587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 74.257271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 74.297900] Lustre: lustre-MDT0000: new disk, initializing [ 74.361779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 74.379108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 74.456926] mount.lustre (7203) used greatest stack depth: 9600 bytes left [ 76.295983] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 85.066460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 85.109045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 85.167277] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 85.186228] Lustre: 8363:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 85.204358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 85.207237] Lustre: Skipped 1 previous similar message [ 85.246620] Lustre: lustre-MDT0001: new disk, initializing [ 85.294995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 85.330445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 85.345547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 87.124313] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 90.432147] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 95.205321] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 95.216184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 95.255835] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 95.261873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 95.386031] Lustre: 9569:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 95.402618] Lustre: lustre-OST0000: new disk, initializing [ 95.406648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 95.454150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 98.181925] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.557267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 101.567324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 101.606185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 105.591926] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 105.610810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 105.653091] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 105.659607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 105.714701] Lustre: 10904:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 105.723913] Lustre: lustre-OST0001: new disk, initializing [ 105.734074] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 105.771307] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 108.178161] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 108.304456] random: crng init done [ 109.737365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 109.745621] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 109.781111] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 115.016910] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 121.134725] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 128.732456] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing check_logdir /tmp/testlogs/ [ 130.208064] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing yml_node [ 132.166983] Lustre: DEBUG MARKER: Client: 2.16.54.69 [ 133.401835] Lustre: DEBUG MARKER: MDS: 2.16.54.69 [ 134.708381] Lustre: DEBUG MARKER: OSS: 2.16.54.69 [ 135.567694] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Thu May 1 20:55:38 EDT 2025 [ 145.580038] Lustre: DEBUG MARKER: - need mds1 <= 2.14.55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 146.431479] Lustre: DEBUG MARKER: - need mds1 < v2_14_55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 147.210319] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42c 42b 118c 118d 407 119i 851 411b [ 147.976958] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 [ 148.870018] Lustre: DEBUG MARKER: === sanity: start setup 20:55:51 (1746147351) === [ 150.832094] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing check_config_client /mnt/lustre [ 162.014431] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 163.983546] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 165.645138] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.745379] Lustre: DEBUG MARKER: === sanity: finish setup 20:56:11 (1746147371) === [ 172.229971] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 20:56:14 (1746147374) [ 173.894274] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 174.692178] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 20:56:17 (1746147377) [ 178.225321] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 20:56:20 (1746147380) [ 201.031595] hrtimer: interrupt took 3851471 ns [ 272.827990] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 20:57:55 (1746147475) [ 276.309201] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 20:57:58 (1746147478) [ 277.024166] Lustre: *** cfs_fail_loc=15b, val=0*** [ 277.028611] Lustre: *** cfs_fail_loc=15b, val=0*** [ 280.781603] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 20:58:03 (1746147483) [ 285.217492] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 20:58:07 (1746147487) [ 285.780931] Lustre: *** cfs_fail_loc=19a, val=0*** [ 286.291703] Lustre: *** cfs_fail_loc=19a, val=0*** [ 287.354375] Lustre: *** cfs_fail_loc=19a, val=0*** [ 287.356226] Lustre: Skipped 1 previous similar message [ 289.645099] Lustre: *** cfs_fail_loc=19a, val=0*** [ 289.647386] Lustre: Skipped 4 previous similar messages [ 293.753141] Lustre: *** cfs_fail_loc=19a, val=0*** [ 293.754602] Lustre: Skipped 9 previous similar messages [ 301.836693] Lustre: *** cfs_fail_loc=19a, val=0*** [ 301.839164] Lustre: Skipped 22 previous similar messages [ 317.845884] Lustre: *** cfs_fail_loc=19a, val=0*** [ 317.848952] Lustre: Skipped 39 previous similar messages [ 332.162464] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 20:58:54 (1746147534) [ 332.609130] Lustre: *** cfs_fail_loc=188, val=0*** [ 334.138032] Lustre: *** cfs_fail_loc=189, val=0*** [ 338.154247] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 338.985684] Lustre: DEBUG MARKER: == sanity test 60j: llog_reader reports corruptions ====== 20:59:01 (1746147541) [ 342.012775] Lustre: lustre-MDD0000: changelog on [ 343.249254] Lustre: lustre-MDD0001: changelog on [ 347.488463] Lustre: DEBUG MARKER: SKIP: sanity test_60j path oi.1/0x1:0xb:0x0 is not in 'O/1/d/' format [ 348.645724] Lustre: lustre-MDD0001: changelog off [ 349.687767] Lustre: lustre-MDD0000: changelog off [ 350.770012] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 20:59:13 (1746147553) [ 353.680452] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 20:59:16 (1746147556) [ 356.973456] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 20:59:19 (1746147559) [ 421.306481] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 21:00:23 (1746147623) [ 428.959284] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 21:00:31 (1746147631) [ 433.262343] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 434.171310] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 21:00:36 (1746147636) [ 438.180464] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 21:00:40 (1746147640) [ 463.324543] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 21:01:05 (1746147665) [ 464.834718] Lustre: *** cfs_fail_loc=725, val=0*** [ 468.046337] Lustre: *** cfs_fail_loc=725, val=0*** [ 471.718631] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 21:01:14 (1746147674) [ 477.360672] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 21:01:19 (1746147679) [ 482.955631] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 21:01:25 (1746147685) [ 495.665405] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 21:01:38 (1746147698) [ 499.010994] Lustre: *** cfs_fail_loc=513, val=17*** [ 499.014313] LustreError: 17255:0:(service.c:2266:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1830968086254208 [ 499.836466] Lustre: Failing over lustre-OST0000 [ 499.867884] LustreError: 32041:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 499.945351] Lustre: server umount lustre-OST0000 complete [ 500.381033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.395222] Lustre: Skipped 1 previous similar message [ 505.386151] LustreError: 17254:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.401324] LustreError: 17254:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 508.619625] LustreError: 17254:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.395217] LustreError: 17254:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.417743] LustreError: 17254:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 513.627168] LustreError: 17254:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.925063] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 513.931763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 513.979153] Lustre: 32580:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 514.036076] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 515.985582] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 516.506097] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [ 517.367100] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 517.367788] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [ 520.915104] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 521.792081] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 527.260227] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 21:02:10 (1746147730) [ 530.299284] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 21:02:12 (1746147732) [ 533.492289] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 21:02:16 (1746147736) [ 536.290062] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 21:02:19 (1746147739) [ 539.306265] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 21:02:21 (1746147741) [ 542.251794] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 21:02:25 (1746147745) [ 544.868827] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 21:02:27 (1746147747) [ 548.453045] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 21:02:31 (1746147751) [ 552.153067] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 21:02:34 (1746147754) [ 556.156977] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 21:02:38 (1746147758) [ 559.921915] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 21:02:42 (1746147762) [ 560.760889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.773013] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 560.779013] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [ 560.783983] Lustre: Skipped 1 previous similar message [ 561.551318] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 561.555343] Lustre: Skipped 1 previous similar message [ 561.958053] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 561.963174] Lustre: Skipped 2 previous similar messages [ 561.966582] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [ 561.970913] Lustre: Skipped 2 previous similar messages [ 571.282816] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 579.681498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.688646] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 579.694102] Lustre: Skipped 1 previous similar message [ 579.696168] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 579.703146] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [ 581.483694] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 581.602601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 583.088419] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 583.194935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 592.287366] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 599.420747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.431066] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 599.435412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 599.441334] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [ 601.002111] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 601.110490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 602.382023] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 602.453752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 609.616980] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 614.862390] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.871445] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 614.878074] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 614.884728] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [ 615.979397] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 616.045614] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 617.190457] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 617.257000] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 624.163117] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 630.866825] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.874108] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 630.881954] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 630.887081] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [ 632.206708] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 632.298933] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 633.736214] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 633.804089] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 636.291549] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 21:03:59 (1746147839) [ 639.949751] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 21:04:02 (1746147842) [ 643.151320] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 21:04:05 (1746147845) [ 660.689272] Lustre: DEBUG MARKER: == sanity test 65o: pool inheritance for mdt component === 21:04:23 (1746147863) [ 676.207441] Lustre: DEBUG MARKER: == sanity test 65p: setstripe with yaml file and huge number ========================================================== 21:04:38 (1746147878) [ 679.467962] Lustre: DEBUG MARKER: == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 21:04:42 (1746147882) [ 682.342736] Lustre: DEBUG MARKER: == sanity test 65r: prevent all-zero offsets ============= 21:04:45 (1746147885) [ 685.205489] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 21:04:47 (1746147887) [ 689.847490] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 21:04:52 (1746147892) [ 690.543010] Lustre: *** cfs_fail_loc=217, val=0*** [ 690.545304] Lustre: Skipped 1 previous similar message [ 691.557433] Lustre: *** cfs_fail_loc=217, val=0*** [ 694.522493] Lustre: DEBUG MARKER: == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 21:04:57 (1746147897) [ 700.560442] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 701.191539] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 21:05:04 (1746147904) [ 704.126202] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 21:05:06 (1746147906) [ 708.383298] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 21:05:10 (1746147910) [ 737.528679] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 21:05:40 (1746147940) [ 740.280591] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 21:05:43 (1746147943) [ 743.383187] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 21:05:46 (1746147946) [ 745.848210] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 21:05:48 (1746147948) [ 782.401938] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 21:06:25 (1746147985) [ 809.657112] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 21:06:52 (1746148012) [ 812.878476] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 21:06:55 (1746148015) [ 813.195952] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.202.30@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3830 extent [0-4194303]: client csum a7362cd8, server csum a7362cd7 [ 815.003290] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 816.244275] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.30@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum df006b69 (type 1), server csum 2ccd497d (type 1) [ 817.221835] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 818.432472] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.30@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum dd4e1822 (type 2), server csum 4e9a1996 (type 2) [ 819.670017] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 820.877938] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.30@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum a4dc3880 (type 4), server csum a7362cd7 (type 4) [ 822.068833] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 823.290533] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.30@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum 63c1e6be (type 10), server csum 22c5e649 (type 10) [ 824.542420] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 827.012587] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 828.291931] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.30@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3830 extent [0-4194303], client returned csum bf2dd087 (type 40), server csum 220d188 (type 40) [ 828.318102] LustreError: Skipped 1 previous similar message [ 829.737163] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 831.983986] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 834.332065] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 21:07:17 (1746148037) [ 837.324729] Lustre: 9580:0:(tgt_handler.c:1935:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000407:0xc20:0x0]:[0-1048575]-61e197fc-401af7b5 [ 837.332523] LustreError: dumping log to /tmp/lustre-log.1746148041.9580 [ 837.389185] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.30@tcp inode [0x200000407:0xc20:0x0] object 0x280000401:3839 extent [0-1048575], client returned csum 61e197fc (type 4), server csum 401af7b5 (type 4) [ 837.398214] LustreError: Skipped 1 previous similar message [ 846.757297] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 21:07:29 (1746148049) [ 846.885726] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.30@tcp inode [0x200000407:0xc22:0x0] object 0x280000401:3840 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 851.547283] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 21:07:34 (1746148054) [ 852.119271] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 852.226878] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.202.30@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3831 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 855.383878] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.202.30@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3831 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 855.397942] LustreError: Skipped 3 previous similar messages [ 862.591144] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.202.30@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3831 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 862.600024] LustreError: Skipped 4 previous similar messages [ 874.274071] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.202.30@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3831 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 874.285307] LustreError: Skipped 2 previous similar messages [ 898.309688] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.202.30@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3831 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 898.315701] LustreError: Skipped 5 previous similar messages [ 909.283725] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 931.583794] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.202.30@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3831 extent [0-4194303]: client csum 7d33b9a0, server csum 7d33b99f [ 931.595624] LustreError: Skipped 16 previous similar messages [ 966.803625] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 995.642803] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.202.30@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3831 extent [4194304-8388607]: client csum de2bf73f, server csum de2bf73e [ 995.654380] LustreError: Skipped 22 previous similar messages [ 1023.634292] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1080.153735] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1125.918861] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.202.30@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3831 extent [4194304-8388607]: client csum fec05011, server csum fec05010 [ 1125.931045] LustreError: Skipped 47 previous similar messages [ 1136.724831] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1192.395271] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1248.778873] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1252.342401] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 21:14:14 (1746148454) [ 1253.234593] Lustre: *** cfs_fail_loc=21a, val=0*** [ 1254.982860] Lustre: *** cfs_fail_loc=21b, val=0*** [ 1260.296649] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 21:14:22 (1746148462) [ 1260.826542] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1262.050560] Lustre: Modifying parameter lustre.osc.lustre*.checksums in log params [ 1265.140755] Lustre: Disabling parameter lustre.osc.lustre*.checksums in log params [ 1269.793136] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1271.727364] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 21:14:34 (1746148474) [ 1272.773839] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 1273.934455] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1276.037191] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid 50 [ 1279.080350] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid in IDLE state after 2 sec [ 1280.982293] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid 50 [ 1281.738706] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid in FULL state after 0 sec [ 1282.387342] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1283.898643] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid 50 [ 1293.978072] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid in IDLE state after 9 sec [ 1295.450828] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid 50 [ 1296.072380] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid in FULL state after 0 sec [ 1296.829714] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1298.605968] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid 50 [ 1308.496264] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid in IDLE state after 9 sec [ 1310.241742] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid 50 [ 1310.990326] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid in FULL state after 0 sec [ 1311.825964] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1313.381930] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid 50 [ 1323.416408] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid in IDLE state after 9 sec [ 1325.233970] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid 50 [ 1325.959584] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid in FULL state after 0 sec [ 1326.944567] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1328.905069] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid 50 [ 1338.973874] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid in IDLE state after 9 sec [ 1340.727317] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid 50 [ 1341.491056] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid in FULL state after 0 sec [ 1342.212643] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1343.720870] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid 50 [ 1353.763616] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid in IDLE state after 9 sec [ 1355.446140] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid 50 [ 1356.186008] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid in FULL state after 0 sec [ 1356.861479] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1358.509317] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid 50 [ 1368.660207] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid in IDLE state after 9 sec [ 1370.586947] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid 50 [ 1371.277290] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800833f5000.ost_server_uuid in FULL state after 0 sec [ 1374.827204] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1375.516329] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 21:16:18 (1746148578) [ 1378.635471] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 21:16:21 (1746148581) [ 1379.633132] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1380.350660] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1381.075827] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1381.774206] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1384.561671] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1385.303051] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 21:16:28 (1746148588) [ 1390.870375] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 21:16:33 (1746148593) [ 1393.937456] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check =========== 21:16:36 (1746148596) [ 1408.911783] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 21:16:51 (1746148611) [ 1413.921838] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 21:16:56 (1746148616) [ 1414.722870] Lustre: *** cfs_fail_loc=228, val=0*** [ 1418.164033] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 21:17:00 (1746148620) [ 1418.766176] Lustre: *** cfs_fail_loc=228, val=0*** [ 1422.601357] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 21:17:05 (1746148625) [ 1437.872961] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ========================================================== 21:17:20 (1746148640) [ 1441.761112] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 21:17:24 (1746148644) [ 1546.083005] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 21:19:08 (1746148748) [ 1562.484542] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 21:19:25 (1746148765) [ 1599.654879] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 21:20:02 (1746148802) [ 1778.167031] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 21:23:00 (1746148980) [ 1818.055113] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 21:23:40 (1746149020) [ 1822.492982] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 21:23:45 (1746149025) [ 1884.129330] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 21:24:46 (1746149086) [ 1894.185270] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 21:24:56 (1746149096) [ 1899.096326] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 21:25:01 (1746149101) [ 1930.837816] Lustre: DEBUG MARKER: == sanity test 101m: read ahead for small file and last stripe of the file ========================================================== 21:25:33 (1746149133) [ 1937.364834] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 21:25:40 (1746149140) [ 1940.935715] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 21:25:43 (1746149143) [ 1946.932195] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 21:25:49 (1746149149) [ 1950.651765] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 21:25:53 (1746149153) [ 1974.474213] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 21:26:17 (1746149177) [ 1996.097103] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 21:26:38 (1746149198) [ 1996.992888] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 1997.850416] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 1998.684018] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 1999.372835] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2002.629723] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 21:26:45 (1746149205) [ 2003.925400] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2004.844654] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 2005.672923] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 2006.550367] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2007.350223] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2010.377221] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 21:26:53 (1746149213) [ 2013.173691] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 21:26:55 (1746149215) [ 2029.806798] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 21:27:12 (1746149232) [ 2034.045924] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 21:27:16 (1746149236) [ 2037.793376] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 21:27:20 (1746149240) [ 2041.637286] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 21:27:24 (1746149244) [ 2045.942378] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 21:27:28 (1746149248) [ 2049.871570] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 21:27:32 (1746149252) [ 2053.339904] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 21:27:35 (1746149255) [ 2057.369990] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 21:27:39 (1746149259) [ 2061.282196] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 21:27:43 (1746149263) [ 2065.378797] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 21:27:47 (1746149267) [ 2203.599430] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 21:30:06 (1746149406) [ 2296.922660] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 21:31:39 (1746149499) [ 2301.654746] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 21:31:44 (1746149504) [ 2436.175731] Lustre: lustre-MDT0000: Client 92c09770-7629-4eec-8d67-ae794b33425b (at 192.168.202.30@tcp) reconnecting [ 2707.970246] Lustre: lustre-MDT0000: Client 92c09770-7629-4eec-8d67-ae794b33425b (at 192.168.202.30@tcp) reconnecting [ 2765.275627] Lustre: lustre-MDT0000: Client 92c09770-7629-4eec-8d67-ae794b33425b (at 192.168.202.30@tcp) reconnecting [ 2806.216868] Lustre: lustre-MDT0000: Client 92c09770-7629-4eec-8d67-ae794b33425b (at 192.168.202.30@tcp) reconnecting [ 2830.552842] Lustre: lustre-MDT0000: Client 92c09770-7629-4eec-8d67-ae794b33425b (at 192.168.202.30@tcp) reconnecting [ 2894.324083] Lustre: lustre-MDT0000: Client 92c09770-7629-4eec-8d67-ae794b33425b (at 192.168.202.30@tcp) reconnecting [ 2894.332908] Lustre: Skipped 1 previous similar message [ 2965.793843] Lustre: lustre-MDT0000: Client 92c09770-7629-4eec-8d67-ae794b33425b (at 192.168.202.30@tcp) reconnecting [ 2996.303585] Lustre: 7226:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 30372 > trans_max 3200 [ 2996.309390] Lustre: 7226:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 2996.313638] Lustre: 7226:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28148/0 [ 2996.326468] Lustre: 7226:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 2996.330917] Lustre: 7226:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 2996.338322] Lustre: 7226:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 2996.343053] CPU: 0 PID: 7226 Comm: mdt00_000 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 2996.348414] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 2996.353227] Call Trace: [ 2996.354228] [] dump_stack+0x19/0x1b [ 2996.357425] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 2996.363525] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 2996.367668] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 2996.370566] [] lod_trans_start+0x7f/0x2e0 [lod] [ 2996.372857] [] mdd_trans_start+0x14/0x20 [mdd] [ 2996.375488] [] mdd_unlink+0x617/0xea0 [mdd] [ 2996.377786] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 2996.382326] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 2996.385899] [] mdt_reint_rec+0x87/0x240 [mdt] [ 2996.389514] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 2996.393086] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 2996.396922] [] mdt_reint+0x67/0x150 [mdt] [ 2996.399812] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 2996.403769] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 2996.407319] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 2996.410046] [] ? put_prev_entity+0x31/0x400 [ 2996.412127] [] ? do_raw_spin_unlock+0x49/0x90 [ 2996.415135] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 2996.417160] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 2996.421979] [] kthread+0xe4/0xf0 [ 2996.424123] [] ? kthread_create_on_node+0x140/0x140 [ 2996.426647] [] ret_from_fork_nospec_begin+0x7/0x21 [ 2996.428842] [] ? kthread_create_on_node+0x140/0x140 [ 2999.497705] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 21:43:22 (1746150202) [ 3001.240211] Lustre: lustre-MDD0000: changelog on [ 3002.689713] Lustre: lustre-MDD0001: changelog on [ 3006.845138] Lustre: lustre-MDD0001: changelog off [ 3008.197154] Lustre: lustre-MDD0000: changelog off [ 3009.540645] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 21:43:32 (1746150212) [ 3011.530196] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012a68e000.ost_server_uuid 50 [ 3012.264234] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a68e000.ost_server_uuid in FULL state after 0 sec [ 3016.059355] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 21:43:38 (1746150218) [ 3019.685488] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 21:43:42 (1746150222) [ 3020.400575] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [ 3021.657221] Lustre: DEBUG MARKER: == sanity test 104d: runas -u 500 -g 500 lctl dl test ==== 21:43:43 (1746150223) [ 3025.394134] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 21:43:47 (1746150227) [ 3029.003100] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 21:43:51 (1746150231) [ 3032.902880] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 21:43:55 (1746150235) [ 3037.505778] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 21:43:59 (1746150239) [ 3052.609325] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 21:44:14 (1746150254) [ 3057.469423] Lustre: DEBUG MARKER: == sanity test 105f: Enqueue same range flocks =========== 21:44:19 (1746150259) [ 3062.482683] Lustre: DEBUG MARKER: == sanity test 105g: ldlm_lock_debug stack test ========== 21:44:25 (1746150265) [ 3069.083716] Lustre: DEBUG MARKER: == sanity test 105h: Flock functional verify ============= 21:44:31 (1746150271) [ 3073.771569] Lustre: DEBUG MARKER: == sanity test 105i: Flock deadlock verify =============== 21:44:36 (1746150276) [ 3082.965902] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 21:44:45 (1746150285) [ 3087.210067] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 21:44:49 (1746150289) [ 3093.671726] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 21:44:56 (1746150296) [ 3098.009452] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 21:45:00 (1746150300) [ 3247.507549] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 21:47:30 (1746150450) [ 3248.884931] Lustre: *** cfs_fail_loc=147, val=0*** [ 3253.750428] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 21:47:36 (1746150456) [ 3257.270297] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 21:47:39 (1746150459) [ 3260.776224] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 21:47:43 (1746150463) [ 3261.697914] Lustre: *** cfs_fail_loc=217, val=0*** [ 3261.700595] Lustre: Skipped 19 previous similar messages [ 3265.501025] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 3266.343585] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 3267.349448] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 21:47:49 (1746150469) [ 3271.531489] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 21:47:53 (1746150473) [ 3274.814747] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 21:47:57 (1746150477) [ 3275.674047] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3276.700128] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3279.387997] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3282.412145] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3286.425616] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3290.584353] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 21:48:13 (1746150493) [ 3294.639704] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3294.641544] Lustre: Skipped 2 previous similar messages [ 3301.805579] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 21:48:24 (1746150504) [ 3306.579801] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 21:48:29 (1746150509) [ 3311.073676] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3311.079209] Lustre: Skipped 24 previous similar messages [ 3318.976716] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 21:48:41 (1746150521) [ 3322.720797] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 21:48:45 (1746150525) [ 3326.074852] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 21:48:48 (1746150528) [ 3329.370656] LustreError: 12027:0:(ofd_dev.c:1805:ofd_statfs_hdl()) cfs_fail_timeout id 242 sleeping for 10000ms [ 3329.478577] LustreError: 12027:0:(ofd_dev.c:1805:ofd_statfs_hdl()) cfs_fail_timeout interrupted [ 3332.350381] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 21:48:55 (1746150535) [ 3335.367929] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 21:48:57 (1746150537) [ 3338.954391] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 21:49:01 (1746150541) [ 3342.389884] Lustre: DEBUG MARKER: == sanity test 119e: Basic tests of dio read and write at various sizes ========================================================== 21:49:04 (1746150544) [ 3344.994251] Lustre: DEBUG MARKER: sanity test_119e: @@@@@@ FAIL: trivial unaligned dio failed [ 3349.728177] Lustre: DEBUG MARKER: == sanity test 119f: dio vs dio race ===================== 21:49:12 (1746150552) [ 3360.150546] Lustre: DEBUG MARKER: sanity test_119f: @@@@@@ FAIL: dio copy 1 w/bsize 1049600 failed: 1 [ 3364.515055] Lustre: DEBUG MARKER: == sanity test 119g: dio vs buffered I/O race ============ 21:49:27 (1746150567) [ 3378.054599] Lustre: DEBUG MARKER: sanity test_119g: @@@@@@ FAIL: dio copy 1 w/bsize 1049600 failed: 1 [ 3382.652983] Lustre: DEBUG MARKER: == sanity test 119h: basic tests of memory unaligned dio ========================================================== 21:49:45 (1746150585) [ 3384.301432] Lustre: DEBUG MARKER: sanity test_119h: @@@@@@ FAIL: multiop memory unaligned write failed, 1044480 [ 3388.124470] Lustre: DEBUG MARKER: SKIP: sanity test_119i skipping ALWAYS excluded test 119i [ 3388.984235] Lustre: DEBUG MARKER: == sanity test 119j: basic tests of hybrid IO switching == 21:49:51 (1746150591) [ 3389.780063] Lustre: DEBUG MARKER: SKIP: sanity test_119j needs kernel > 4.5.0 for ki_flags support [ 3390.605004] Lustre: DEBUG MARKER: == sanity test 119m: Test DIO readv/writev: exercise iter duplication ========================================================== 21:49:53 (1746150593) [ 3394.065696] Lustre: DEBUG MARKER: == sanity test 119n: Test Unaligned DIO readv() and writev() with unpatched ZFS ========================================================== 21:49:56 (1746150596) [ 3394.969359] Lustre: DEBUG MARKER: SKIP: sanity test_119n need ZFS server without unaligned_dio support [ 3395.937685] Lustre: DEBUG MARKER: == sanity test 119o: Test Unaligned DIO readv() and writev() with unpatched servers ========================================================== 21:49:58 (1746150598) [ 3396.926271] Lustre: DEBUG MARKER: SKIP: sanity test_119o need ldiskfs without unaligned_dio support. [ 3397.953792] Lustre: DEBUG MARKER: == sanity test 119p: Test Unaligned DIO readv() and writev() with patched servers ========================================================== 21:50:00 (1746150600) [ 3398.840900] Lustre: DEBUG MARKER: sanity test_119p: @@@@@@ FAIL: DIO unaligned writev test failed [ 3401.787992] Lustre: DEBUG MARKER: == sanity test 119q: Test patchded Unaligned DIO readv() and writev() ========================================================== 21:50:04 (1746150604) [ 3402.740639] Lustre: DEBUG MARKER: sanity test_119q: @@@@@@ FAIL: DIO unaligned writev test failed: 69631 [ 3405.652168] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 21:50:08 (1746150608) [ 3410.981269] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 21:50:13 (1746150613) [ 3416.134948] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 21:50:18 (1746150618) [ 3421.657155] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 21:50:24 (1746150624) [ 3426.382003] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 21:50:29 (1746150629) [ 3438.074702] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 21:50:40 (1746150640) [ 3450.495010] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 21:50:53 (1746150653) [ 3695.272706] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 21:54:57 (1746150897) [ 3699.249457] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 21:55:01 (1746150901) [ 3704.196086] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 3705.624544] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 3727.198528] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 13 sec [ 3730.350332] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 3 sec [ 3783.640942] Lustre: DEBUG MARKER: ls -l 3000 files without statahead: 38 sec [ 3790.924495] Lustre: DEBUG MARKER: ls -l 3000 files with statahead: 6 sec [ 3791.742857] Lustre: DEBUG MARKER: ls -l done [ 3819.742721] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity: 27 seconds [ 3823.174376] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 21:57:05 (1746151025) [ 3823.823855] Lustre: DEBUG MARKER: SKIP: sanity test_123ab Test must be statx() syscall supported [ 3824.523318] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 21:57:07 (1746151027) [ 3825.141584] Lustre: DEBUG MARKER: SKIP: sanity test_123ac Test must be statx() syscall supported [ 3825.904387] Lustre: DEBUG MARKER: == sanity test 123ad: Verify batching statahead works correctly ========================================================== 21:57:08 (1746151028) [ 3830.492794] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 2 sec [ 3831.839019] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 3850.747401] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 11 sec [ 3855.417902] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 4 sec [ 3919.459529] Lustre: DEBUG MARKER: ls -l 4000 files without statahead: 43 sec [ 3932.991961] Lustre: DEBUG MARKER: ls -l 4000 files with statahead: 12 sec [ 3933.814245] Lustre: DEBUG MARKER: ls -l done [ 3968.159876] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 33 seconds [ 4198.656502] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 4199.803026] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 0 sec [ 4216.453077] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 10 sec [ 4219.187527] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 2 sec [ 4283.711627] Lustre: DEBUG MARKER: ls -l 5000 files without statahead: 44 sec [ 4292.142564] Lustre: DEBUG MARKER: ls -l 5000 files with statahead: 6 sec [ 4292.740836] Lustre: DEBUG MARKER: ls -l done [ 4331.629812] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 38 seconds [ 4512.600498] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 22:08:35 (1746151715) [ 4522.085981] Lustre: DEBUG MARKER: ls done [ 4532.978954] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 22:08:55 (1746151735) [ 4538.491491] Lustre: DEBUG MARKER: == sanity test 123d: Statahead on striped directories works correctly ========================================================== 22:09:01 (1746151741) [ 4548.656374] Lustre: DEBUG MARKER: == sanity test 123e: statahead with large wide striping == 22:09:11 (1746151751) [ 4608.968008] Lustre: 21225:0:(mdt_handler.c:4672:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 4664.998986] Lustre: DEBUG MARKER: == sanity test 123f: Retry mechanism with large wide striping files ========================================================== 22:11:07 (1746151867) [ 4730.871665] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4730.900348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4853.757074] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4854.493198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4933.382454] Lustre: 21225:0:(mdt_handler.c:4672:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 5086.057667] Lustre: 7228:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 26652 > trans_max 3200 [ 5086.065142] Lustre: 7228:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 5086.073894] Lustre: 7228:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1755/1755/0, xattr_set: 2632/24676/0 [ 5086.080153] Lustre: 7228:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 5086.089940] Lustre: 7228:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 5086.100389] Lustre: 7228:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 5086.108987] CPU: 3 PID: 7228 Comm: mdt00_002 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 5086.115907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 5086.124187] Call Trace: [ 5086.125646] [] dump_stack+0x19/0x1b [ 5086.127126] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 5086.129987] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 5086.132306] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 5086.134018] [] lod_trans_start+0x7f/0x2e0 [lod] [ 5086.136324] [] mdd_trans_start+0x14/0x20 [mdd] [ 5086.138395] [] mdd_unlink+0x617/0xea0 [mdd] [ 5086.140668] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 5086.143340] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 5086.149618] [] mdt_reint_rec+0x87/0x240 [mdt] [ 5086.151617] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 5086.158569] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 5086.163889] [] mdt_reint+0x67/0x150 [mdt] [ 5086.172334] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 5086.175553] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 5086.182780] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 5086.184913] [] ? put_prev_entity+0x31/0x400 [ 5086.187188] [] ? do_raw_spin_unlock+0x49/0x90 [ 5086.189375] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 5086.191894] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 5086.199342] [] kthread+0xe4/0xf0 [ 5086.200999] [] ? kthread_create_on_node+0x140/0x140 [ 5086.207638] [] ret_from_fork_nospec_begin+0x7/0x21 [ 5086.210015] [] ? kthread_create_on_node+0x140/0x140 [ 5092.384038] Lustre: DEBUG MARKER: == sanity test 123g: Test for stat-ahead advise ========== 22:18:14 (1746152294) [ 5117.755299] Lustre: DEBUG MARKER: == sanity test 123h: Verify statahead work with the fname pattern via du ========================================================== 22:18:40 (1746152320) [ 5976.840968] Lustre: DEBUG MARKER: == sanity test 123i: Verify statahead work with the fname indexing pattern ========================================================== 22:32:59 (1746153179) [ 6066.776389] Lustre: DEBUG MARKER: == sanity test 123j: -ENOENT error from batched statahead be handled correctly ========================================================== 22:34:29 (1746153269) [ 6071.244073] Lustre: DEBUG MARKER: == sanity test 123k: Verify statahead work with mdtest shared stat() mode ========================================================== 22:34:33 (1746153273) [ 6071.938828] Lustre: DEBUG MARKER: SKIP: sanity test_123k mdtest not found [ 6073.015735] Lustre: DEBUG MARKER: == sanity test 123l: Avoid panic when revalidate a local cached entry ========================================================== 22:34:35 (1746153275) [ 6147.091086] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 22:35:49 (1746153349) [ 6147.955254] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 6166.860054] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff88012d48a800 [ 6167.667547] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff88012d48a800 [ 6168.406967] Lustre: DEBUG MARKER: LRU=1004 [ 6169.214689] Lustre: DEBUG MARKER: LIMIT=46624 [ 6169.958731] Lustre: DEBUG MARKER: LVF=5572500 [ 6170.722908] Lustre: DEBUG MARKER: OLD_LVF=100 [ 6171.421770] Lustre: DEBUG MARKER: Sleep 50 sec [ 6222.472702] Lustre: DEBUG MARKER: Dropped 425 locks in 50s [ 6223.326834] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 6240.368108] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 22:37:22 (1746153442) [ 6297.776116] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 6384.982083] Lustre: DEBUG MARKER: ls -la time: 86 seconds [ 6385.788867] Lustre: DEBUG MARKER: lru_size = 400 [ 6478.634079] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 6513.975204] Lustre: DEBUG MARKER: ls -la time: 33 seconds [ 6514.680176] Lustre: DEBUG MARKER: lru_size = 4052 [ 6515.410915] Lustre: DEBUG MARKER: ls -la is 61% faster with lru resize enabled [ 6552.726886] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 22:42:35 (1746153755) [ 6580.718553] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize disabled ========================================================== 22:43:03 (1746153783) [ 6608.179626] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 22:43:30 (1746153810) [ 6612.297693] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 22:43:34 (1746153814) [ 6616.324145] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 22:43:38 (1746153818) [ 6621.537509] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 22:43:43 (1746153823) [ 6626.323649] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [ 6649.534409] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 22:44:11 (1746153851) [ 6654.114169] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 22:44:16 (1746153856) [ 6665.888413] Lustre: 9154:0:(osd_handler.c:600:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513277, FID: [0x20000040b:0x830c:0x0]) is approaching max size limit [ 6667.538993] Lustre: 9154:0:(osd_handler.c:600:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513277, FID: [0x20000040b:0x830c:0x0]) is approaching max size limit [ 6674.324184] Lustre: 7228:0:(osd_handler.c:596:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513277, FID: [0x20000040b:0x830c:0x0]) has reached max size limit [ 6693.226558] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 22:44:55 (1746153895) [ 6697.742889] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 22:45:00 (1746153900) [ 6702.568131] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 22:45:05 (1746153905) [ 6706.785652] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 22:45:09 (1746153909) [ 6707.778146] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [ 6709.124987] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 22:45:11 (1746153911) [ 6746.734042] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 22:45:49 (1746153949) [ 6750.240968] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 22:45:52 (1746153952) [ 6764.657151] Lustre: 9155:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 6372 > trans_max 3200 [ 6764.671676] Lustre: 9155:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 6764.677837] Lustre: 9155:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 403/403/0, xattr_set: 604/5748/0 [ 6764.682009] Lustre: 9155:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 6764.688865] Lustre: 9155:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 6764.697707] Lustre: 9155:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 6764.704428] CPU: 3 PID: 9155 Comm: mdt00_008 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 6764.708795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 6764.714398] Call Trace: [ 6764.715395] [] dump_stack+0x19/0x1b [ 6764.719516] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 6764.723981] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 6764.727416] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 6764.730596] [] lod_trans_start+0x7f/0x2e0 [lod] [ 6764.733775] [] mdd_trans_start+0x14/0x20 [mdd] [ 6764.736657] [] mdd_unlink+0x617/0xea0 [mdd] [ 6764.739533] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 6764.742992] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 6764.746027] [] mdt_reint_rec+0x87/0x240 [mdt] [ 6764.749786] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 6764.753250] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 6764.755863] [] mdt_reint+0x67/0x150 [mdt] [ 6764.759364] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 6764.762785] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 6764.766631] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 6764.769836] [] ? put_prev_entity+0x31/0x400 [ 6764.773733] [] ? do_raw_spin_unlock+0x49/0x90 [ 6764.777364] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 6764.779554] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 6764.781971] [] kthread+0xe4/0xf0 [ 6764.786156] [] ? kthread_create_on_node+0x140/0x140 [ 6764.789476] [] ret_from_fork_nospec_begin+0x7/0x21 [ 6764.793078] [] ? kthread_create_on_node+0x140/0x140 [ 6767.753986] Lustre: DEBUG MARKER: == sanity test 130h: FIEMAP deadlock ===================== 22:46:10 (1746153970) [ 6777.666225] Lustre: DEBUG MARKER: == sanity test 130i: FIEMAP (DoM file) =================== 22:46:20 (1746153980) [ 6804.978221] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 22:46:47 (1746154007) [ 6809.026190] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 22:46:51 (1746154011) [ 6813.106852] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 22:46:55 (1746154015) [ 6816.524837] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 22:46:59 (1746154019) [ 6820.328380] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 22:47:02 (1746154022) [ 6824.038435] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 22:47:06 (1746154026) [ 6835.649312] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 22:47:18 (1746154038) [ 6842.384944] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 22:47:24 (1746154044) [ 6863.203632] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 22:47:45 (1746154065) [ 6882.723491] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 22:48:05 (1746154085) [ 6890.027876] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 22:48:12 (1746154092) [ 6902.892227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6902.892888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6902.904439] Lustre: Skipped 3 previous similar messages [ 6906.394144] LustreError: 24167:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6906.398153] LustreError: 24167:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6906.561579] Lustre: server umount lustre-MDT0000 complete [ 6907.897532] LustreError: 6562:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6907.909792] LustreError: 6562:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6911.235295] LustreError: 4326:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746154115 with bad export cookie 1839710779945603665 [ 6911.238551] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6911.256426] LustreError: 4326:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6911.379735] LustreError: 24769:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6911.382214] LustreError: 24769:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6911.540813] Lustre: server umount lustre-MDT0001 complete [ 6927.049666] LustreError: 25364:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6927.054756] LustreError: 25364:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6927.131117] Lustre: server umount lustre-OST0000 complete [ 6928.917678] Lustre: 3763:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746154117/real 1746154117] req@ffff88012b716d80 x1830968148609920/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746154133 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 6928.931608] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6931.958896] LustreError: 25968:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6931.962329] LustreError: 25968:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6932.157203] Lustre: server umount lustre-OST0001 complete [ 6935.475838] device-mapper: core: cleaned up [ 6940.129797] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 6941.442026] Key type lgssc unregistered [ 6941.578370] LNet: 26822:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6942.592289] LNet: Removed LNI 192.168.202.130@tcp [ 6948.977937] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 6949.192368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6949.203547] alg: No test for adler32 (adler32-zlib) [ 6950.072034] Lustre: Lustre: Build Version: 2.16.54_69_g06f2fd3 [ 6950.187893] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 6950.190782] LNet: Accept secure, port 988 [ 6951.781967] Key type lgssc registered [ 6952.268702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6959.197808] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 6961.543061] device-mapper: uevent: version 1.0.3 [ 6961.545202] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 6965.424485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6965.449123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.928722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6968.902199] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [ 6975.334244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6975.412155] Lustre: 30103:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6975.653135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6975.703202] mount.lustre (30103) used greatest stack depth: 9504 bytes left [ 6977.220328] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [ 6978.926158] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 6983.457970] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6983.468091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6983.626089] Lustre: 31230:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6983.754965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6984.744538] LustreError: 31401:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6984.755819] LustreError: 31401:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6984.772410] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:11591 to 0x280000400:11617) [ 6986.292718] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [ 6989.759201] LustreError: 31235:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6991.763763] LustreError: 31401:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6991.776572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:22409 to 0x280000403:22529) [ 6992.263132] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6992.272919] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6992.354828] Lustre: 32428:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6992.474479] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6994.963645] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [ 6997.172149] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:11431 to 0x2c0000400:11457) [ 6997.176553] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23183 to 0x2c0000403:23297) [ 7000.249056] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7002.030561] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 7011.634218] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 22:50:14 (1746154214) [ 7041.265627] LNet: 2726:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 7041.315498] Lustre: DEBUG MARKER:  [ 7041.316916] Lustre: DEBUG MARKER:  [ 7069.613066] LNet: 4341:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 7069.616967] LNet: 4341:0:(debug.c:388:cfs_str2mask()) Skipped 3 previous similar messages [ 7069.644089] Lustre: DEBUG MARKER:  [ 7069.645787] Lustre: DEBUG MARKER:  [ 7075.771344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7075.776462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7075.790546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7077.679464] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.685584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7077.694921] Lustre: Skipped 2 previous similar messages [ 7078.967357] LustreError: 4630:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7079.052956] Lustre: server umount lustre-MDT0000 complete [ 7082.687065] LustreError: 31254:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7082.701887] LustreError: 31254:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 7083.148087] LustreError: 4631:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746154287 with bad export cookie 9445481963822097051 [ 7083.152381] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7083.159879] LustreError: 4631:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7083.254810] LustreError: 5232:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 7083.258315] LustreError: 5232:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7083.419171] Lustre: server umount lustre-MDT0001 complete [ 7096.730617] LustreError: 5824:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 7096.733016] LustreError: 5824:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7096.768469] Lustre: server umount lustre-OST0000 complete [ 7111.048478] LustreError: 6428:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7111.050864] LustreError: 6428:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7111.205371] Lustre: server umount lustre-OST0001 complete [ 7114.583379] device-mapper: core: cleaned up [ 7119.243283] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 7120.421632] Key type lgssc unregistered [ 7120.568876] LNet: 7359:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7121.581626] LNet: Removed LNI 192.168.202.130@tcp [ 7128.147884] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 7128.352100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7128.420073] alg: No test for adler32 (adler32-zlib) [ 7129.273944] Lustre: Lustre: Build Version: 2.16.54_69_g06f2fd3 [ 7129.383047] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 7129.384864] LNet: Accept secure, port 988 [ 7130.932130] Key type lgssc registered [ 7131.439728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7137.857296] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 7140.240019] device-mapper: uevent: version 1.0.3 [ 7140.241835] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 7143.452486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7143.465393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7144.721795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7146.001212] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [ 7150.967450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7151.062811] Lustre: 10639:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7151.288687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7152.857689] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [ 7154.215867] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 7158.755737] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7158.764214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7158.891872] Lustre: 11767:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7159.030576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7161.029787] LustreError: 11774:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7161.042652] LustreError: 11774:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 7161.049935] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:11591 to 0x280000400:11649) [ 7161.792350] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [ 7166.046302] LustreError: 12041:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7167.372036] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 7167.381181] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7167.452018] Lustre: 12965:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7167.584046] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7169.599046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:22409 to 0x280000403:22561) [ 7169.605441] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:11431 to 0x2c0000400:11489) [ 7170.068436] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [ 7170.597968] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23183 to 0x2c0000403:23329) [ 7175.588794] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7177.535953] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 7186.517079] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 22:53:09 (1746154389) [ 7703.007623] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 23:01:45 (1746154905) [ 7712.987552] Lustre: *** cfs_fail_loc=327, val=500*** [ 7735.574524] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 23:02:17 (1746154937) [ 7739.199754] Lustre: *** cfs_fail_loc=328, val=500*** [ 7739.201493] Lustre: Skipped 1025 previous similar messages [ 7740.201708] Lustre: *** cfs_fail_loc=328, val=500*** [ 7740.203435] Lustre: Skipped 155 previous similar messages [ 7741.401822] Lustre: 22441:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff880071658000 x1830975519684352/t0(0) o101->4f88656b-4e5d-4b1a-ac91-d79d672a4453@192.168.202.30@tcp:16/0 lens 648/0 e 0 to 0 dl 1746154956 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7742.438203] Lustre: *** cfs_fail_loc=328, val=500*** [ 7742.440112] Lustre: Skipped 190 previous similar messages [ 7742.447767] Lustre: 22441:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff880097f87480 x1830975519685120/t0(0) o101->4f88656b-4e5d-4b1a-ac91-d79d672a4453@192.168.202.30@tcp:17/0 lens 648/0 e 0 to 0 dl 1746154957 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7743.509959] Lustre: 11792:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8800716abb80 x1830975519686400/t0(0) o101->4f88656b-4e5d-4b1a-ac91-d79d672a4453@192.168.202.30@tcp:18/0 lens 648/0 e 0 to 0 dl 1746154958 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7746.047491] Lustre: 22441:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff880119e2f100 x1830975519726848/t0(0) o101->4f88656b-4e5d-4b1a-ac91-d79d672a4453@192.168.202.30@tcp:21/0 lens 648/0 e 0 to 0 dl 1746154961 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7746.059362] Lustre: 22441:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) Skipped 1 previous similar message [ 7747.082464] Lustre: *** cfs_fail_loc=328, val=500*** [ 7747.085386] Lustre: Skipped 79 previous similar messages [ 7750.120495] Lustre: 11792:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff88011d8ae680 x1830975519730304/t0(0) o101->4f88656b-4e5d-4b1a-ac91-d79d672a4453@192.168.202.30@tcp:25/0 lens 648/0 e 0 to 0 dl 1746154965 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7750.136635] Lustre: 11792:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) Skipped 3 previous similar messages [ 7755.428887] Lustre: *** cfs_fail_loc=328, val=500*** [ 7755.430917] Lustre: Skipped 42 previous similar messages [ 7759.558976] LustreError: 21868:0:(ldlm_lockd.c:3258:lock_reclaim_threshold_mb_store()) Failed to set lock_reclaim_threshold_mb, rc = -22. [ 7768.936675] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [ 7769.824322] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [ 7770.796333] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 23:02:53 (1746154973) [ 7793.246028] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 23:03:15 (1746154995) [ 7812.855102] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 23:03:35 (1746155015) [ 7834.906121] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 23:03:57 (1746155037) [ 7853.528363] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 23:04:16 (1746155056) [ 7864.069081] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 23:04:26 (1746155066) [ 7874.063240] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 23:04:36 (1746155076) [ 7889.405015] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 23:04:51 (1746155091) [ 7899.598349] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 23:05:01 (1746155101) [ 7917.397072] Lustre: DEBUG MARKER: == sanity test 150h: Verify extend fallocate updates the file size ========================================================== 23:05:19 (1746155119) [ 7918.751757] Lustre: DEBUG MARKER: SKIP: sanity test_150h Test must be statx() syscall supported [ 7919.924174] Lustre: DEBUG MARKER: == sanity test 150ia: Verify fallocate zero-range ZERO functionality ========================================================== 23:05:22 (1746155122) [ 7930.003202] Lustre: DEBUG MARKER: == sanity test 150ib: Verify fallocate zero-range PREALLOC functionality ========================================================== 23:05:32 (1746155132) [ 7942.846440] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 23:05:45 (1746155145) [ 7954.334193] bash (31797): drop_caches: 1 [ 7961.833553] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 23:06:04 (1746155164) [ 7965.836924] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 23:06:08 (1746155168) [ 7969.207070] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 23:06:11 (1746155171) [ 7972.276656] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 23:06:15 (1746155175) [ 7975.074435] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 23:06:17 (1746155177) [ 7975.892643] LustreError: 9565:0:(fld_handler.c:244:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 7980.365851] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 23:06:22 (1746155182) [ 7981.235431] LustreError: 11792:0:(fld_handler.c:244:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 7981.240281] LustreError: 11792:0:(fld_handler.c:244:fld_server_lookup()) Skipped 3 previous similar messages [ 7985.676758] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 23:06:28 (1746155188) [ 7989.068436] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 23:06:31 (1746155191) [ 7993.273551] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 23:06:35 (1746155195) [ 7996.663133] Lustre: DEBUG MARKER: == sanity test 154ea: .lustre is not returned by readdir (2) ========================================================== 23:06:39 (1746155199) [ 8035.903040] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 23:07:18 (1746155238) [ 8040.999612] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 23:07:23 (1746155243) [ 8589.357059] Lustre: DEBUG MARKER: == sanity test 154h: Verify interactive path2fid ========= 23:16:32 (1746155792) [ 8591.961380] Lustre: DEBUG MARKER: == sanity test 154i: fid2path for path longer than PATH_MAX ========================================================== 23:16:34 (1746155794) [ 8611.748312] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 23:16:54 (1746155814) [ 8617.432347] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 23:17:00 (1746155820) [ 8623.569149] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 23:17:06 (1746155826) [ 8631.082059] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 23:17:13 (1746155833) [ 8638.316529] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 23:17:20 (1746155840) [ 8665.427146] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 23:17:47 (1746155867) [ 8695.518833] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 23:18:18 (1746155898) [ 8726.484794] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 23:18:48 (1746155928) [ 8757.113652] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 23:19:19 (1746155959) [ 8762.452172] Lustre: DEBUG MARKER: Turn on read and write cache [ 8764.635150] Lustre: DEBUG MARKER: Write data and read it back. [ 8765.682215] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 8768.458017] Lustre: DEBUG MARKER: cache hits: before: 65582, after: 65585 [ 8769.367803] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8770.911991] Lustre: DEBUG MARKER: cache hits:: before: 65585, after: 65588 [ 8771.676087] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [ 8774.389753] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8776.959434] Lustre: DEBUG MARKER: cache hits:: before: 65588, after: 65591 [ 8778.079141] Lustre: DEBUG MARKER: Write data and read it back. [ 8779.061292] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 8781.699012] Lustre: DEBUG MARKER: cache hits:: before: 65591, after: 65594 [ 8782.692096] Lustre: DEBUG MARKER: Turn off read and write cache [ 8785.065111] Lustre: DEBUG MARKER: Write data and read it back [ 8785.934750] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 8788.132450] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65594 [ 8788.948109] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [ 8791.215831] Lustre: DEBUG MARKER: Write data and read it back [ 8792.038483] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 8794.383422] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65594 [ 8795.140171] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8797.367201] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65597 [ 8802.116061] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 23:20:04 (1746156004) [ 8803.584029] Lustre: lustre-MDD0000: changelog on [ 8805.113722] Lustre: lustre-MDD0001: changelog on [ 8815.364719] Lustre: Failing over lustre-MDT0000 [ 8816.418940] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 8816.524340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8816.540985] Lustre: Skipped 2 previous similar messages [ 8821.409711] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 8821.417701] Lustre: Skipped 4 previous similar messages [ 8821.482069] LustreError: 15892:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8821.532281] LustreError: 9565:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8821.548631] LustreError: 9565:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 8821.607650] Lustre: server umount lustre-MDT0000 complete [ 8826.412697] LustreError: 9567:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8826.801899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8826.930810] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8827.160780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8827.185414] Lustre: lustre-MDD0000: changelog on [ 8827.197976] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8829.054927] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [ 8831.421600] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8832.159768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [ 8832.469883] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8832.498108] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:24163 to 0x2c0000403:24193) [ 8832.498563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:23399 to 0x280000403:23425) [ 8832.920213] Lustre: lustre-MDD0000: changelog off [ 8833.850260] Lustre: lustre-MDD0001: changelog off [ 8840.658903] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 23:20:43 (1746156043) [ 8842.082099] Lustre: lustre-MDD0000: changelog on [ 8848.115072] Lustre: lustre-MDD0001: changelog off [ 8851.015557] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 23:20:53 (1746156053) [ 8852.468967] Lustre: lustre-MDD0000: changelog on [ 8852.470488] Lustre: Skipped 1 previous similar message [ 8859.124604] Lustre: lustre-MDD0001: changelog off [ 8859.126802] Lustre: Skipped 1 previous similar message [ 8862.132841] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 23:21:04 (1746156064) [ 8863.604988] Lustre: lustre-MDD0000: changelog on [ 8863.608723] Lustre: Skipped 1 previous similar message [ 8869.060951] Lustre: lustre-MDD0001: changelog off [ 8869.067441] Lustre: Skipped 1 previous similar message [ 8872.346901] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 23:21:14 (1746156074) [ 8879.611319] Lustre: lustre-MDD0001: changelog off [ 8879.613293] Lustre: Skipped 1 previous similar message [ 8882.392217] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 23:21:24 (1746156084) [ 8883.972300] Lustre: lustre-MDD0000: changelog on [ 8883.974097] Lustre: Skipped 3 previous similar messages [ 8888.859676] Lustre: DEBUG MARKER: 1746156091: creating first dirs [ 8904.844391] Lustre: *** cfs_fail_loc=1313, val=3*** [ 8904.846374] Lustre: 9565:0:(mdd_dir.c:911:mdd_changelog_emrg_cleanup()) lustre-MDD0000: changelog has only 3 free catalog entries [ 8904.852381] Lustre: 9565:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 8904.870123] Lustre: 24379:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl7 idle for 19s with 4 unprocessed records [ 8915.396223] Lustre: lustre-MDD0001: changelog off [ 8915.400258] Lustre: Skipped 1 previous similar message [ 8917.735144] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 23:22:00 (1746156120) [ 8918.990675] Lustre: lustre-MDD0000: changelog on [ 8918.992141] Lustre: Skipped 1 previous similar message [ 8929.327119] Lustre: 22441:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 8929.331044] Lustre: 22441:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 8929.338093] Lustre: 27678:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl9 idle for 8s with 4 unprocessed records [ 8929.345894] Lustre: 27678:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 8939.963436] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 23:22:22 (1746156142) [ 8964.735183] Lustre: *** cfs_fail_loc=1316, val=0*** [ 8964.737288] Lustre: 22441:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0001: simulate starting changelog garbage collection [ 8964.743484] Lustre: 22441:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 8964.749433] Lustre: 30966:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0001: force deregister of changelog user cl11 idle for 20s with 3 unprocessed records [ 8964.762295] Lustre: 30966:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 8965.816632] Lustre: Failing over lustre-MDT0001 [ 8965.835362] Lustre: Failing over lustre-MDT0000 [ 8966.654377] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.30@tcp (stopping) [ 8967.401448] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8967.406556] Lustre: Skipped 4 previous similar messages [ 8969.462205] LustreError: 31259:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8969.464577] LustreError: 31259:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8969.496959] Lustre: server umount lustre-MDT0001 complete [ 8969.566564] Lustre: server umount lustre-MDT0000 complete [ 8973.308041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8973.413877] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8973.616354] LustreError: 11783:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8973.627421] LustreError: 11783:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 8973.700061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8973.734942] Lustre: 32507:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 29 to index 30, being cleared now [ 8973.758368] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8975.399633] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [ 8976.692260] LustreError: 11782:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8976.700209] LustreError: 11782:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 8979.644264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8979.713237] Lustre: 963:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8979.804400] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 8979.872061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8979.895906] Lustre: 963:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0001 : orphan changelog records found, starting from index 17 to index 18, being cleared now [ 8979.911570] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8981.225983] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [ 8984.875919] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8986.891669] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [ 8986.897137] Lustre: Skipped 3 previous similar messages [ 8986.899146] Lustre: lustre-MDT0001: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 8986.916049] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:11657 to 0x280000400:11681) [ 8986.917272] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:11496 to 0x2c0000400:11521) [ 8987.329298] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:24163 to 0x2c0000403:24225) [ 8987.329417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:23427 to 0x280000403:23457) [ 8996.134061] Lustre: lustre-MDD0001: changelog off [ 8996.135931] Lustre: Skipped 3 previous similar messages [ 8999.428430] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 23:23:21 (1746156201) [ 9000.975257] Lustre: lustre-MDD0000: changelog on [ 9000.977116] Lustre: Skipped 5 previous similar messages [ 9004.828756] LustreError: 3512:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_race id 1315 sleeping [ 9007.114757] LustreError: 3683:0:(mdd_device.c:1746:mdd_changelog_user_register()) cfs_fail_race id 1315 waking [ 9007.120412] LustreError: 3512:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_fail_race id 1315 awake: rc=2712 [ 9008.078928] LustreError: 3936:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_fail_race id 1315 waking [ 9018.619645] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 23:23:41 (1746156221) [ 9030.886112] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 23:23:53 (1746156233) [ 9034.752683] LustreError: 32517:0:(mdd_dir.c:966:mdd_changelog_store()) cfs_fail_timeout id 15d sleeping for 3000ms [ 9037.760610] LustreError: 32517:0:(mdd_dir.c:966:mdd_changelog_store()) cfs_fail_timeout id 15d awake [ 9048.288063] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 23:24:10 (1746156250) [ 9060.578657] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 23:24:23 (1746156263) [ 9068.419368] LustreError: 32517:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_race id 15f sleeping [ 9070.432061] LustreError: 1580:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_fail_race id 15f waking [ 9070.444470] LustreError: 1580:0:(mdd_device.c:394:llog_changelog_cancel_cb()) Skipped 1 previous similar message [ 9070.456749] LustreError: 32517:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_fail_race id 15f awake: rc=2966 [ 9076.962579] Lustre: lustre-MDD0001: changelog off [ 9076.964501] Lustre: Skipped 11 previous similar messages [ 9079.841985] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 23:24:42 (1746156282) [10068.448640] LustreError: 1584:0:(mdd_device.c:408:llog_changelog_cancel_cb()) cfs_race id 16c sleeping [10070.457810] LustreError: 32517:0:(llog_osd.c:1136:llog_osd_next_block()) cfs_fail_race id 16c waking [10070.464609] LustreError: 1584:0:(mdd_device.c:408:llog_changelog_cancel_cb()) cfs_fail_race id 16c awake: rc=2994 [10076.741134] Lustre: lustre-MDD0001: changelog off [10076.743312] Lustre: Skipped 1 previous similar message [10078.937851] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 23:41:21 (1746157281) [10080.479295] Lustre: lustre-MDD0000: changelog on [10080.483668] Lustre: Skipped 13 previous similar messages [10082.541846] LustreError: 14718:0:(mdd_device.c:1700:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [10083.036794] Lustre: 14782:0:(mdd_device.c:1717:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [10083.495047] LustreError: 14846:0:(mdd_device.c:1709:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [10095.276724] Lustre: lustre-MDD0000: changelog off [10095.278582] Lustre: Skipped 1 previous similar message [10098.909475] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 23:41:41 (1746157301) [10100.589277] Lustre: lustre-MDD0000: changelog on [10100.596434] Lustre: Skipped 1 previous similar message [10105.034671] Lustre: Failing over lustre-MDT0000 [10105.156799] LustreError: 17299:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [10105.159532] LustreError: 17299:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [10105.331316] Lustre: server umount lustre-MDT0000 complete [10107.755172] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.762492] LustreError: 972:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10107.799527] Lustre: Skipped 3 previous similar messages [10110.147618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10110.228638] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10110.409230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10110.441336] Lustre: 17960:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 90153 to index 18446744073709551615, being cleared now [10110.462982] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10111.946689] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [10112.941217] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10115.411593] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [10115.414732] Lustre: Skipped 5 previous similar messages [10115.436187] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10115.442379] Lustre: Skipped 1 previous similar message [10115.477415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:23459 to 0x280000403:23489) [10115.479825] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:24227 to 0x2c0000403:24257) [10121.677136] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 23:42:04 (1746157324) [10127.263619] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records * time ========================================================== 23:42:09 (1746157329) [10137.103160] Lustre: 7094:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [10137.107247] Lustre: 7094:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [10137.113534] Lustre: 20538:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl2 idle for 864008s with 500000004 unprocessed records [10137.122713] Lustre: lustre-MDD0000: changelog off [10137.127120] Lustre: Skipped 3 previous similar messages [10147.391996] Lustre: DEBUG MARKER: == sanity test 160t: changelog garbage collect on lack of space ========================================================== 23:42:29 (1746157349) [10149.266381] Lustre: lustre-MDD0000: changelog on [10149.268014] Lustre: Skipped 4 previous similar messages [10188.403107] Lustre: *** cfs_fail_loc=18c, val=1211180*** [10188.406380] Lustre: Skipped 1 previous similar message [10188.413233] Lustre: 1584:0:(mdd_dir.c:881:mdd_changelog_is_space_safe()) lustre-MDD0000: changelog uses 21MB with 1MB space limit [10188.417836] Lustre: 1584:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [10188.425095] Lustre: 1584:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [10188.431208] Lustre: 23218:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl3-user1 idle for 39s with 7505 unprocessed records [10188.440582] Lustre: 23218:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [10188.969265] Lustre: *** cfs_fail_loc=18c, val=1211180*** [10210.227674] Lustre: lustre-MDD0001: changelog off [10210.229833] Lustre: Skipped 2 previous similar messages [10212.675418] Lustre: DEBUG MARKER: == sanity test 160u: changelog rename record type name and sname strings are correct ========================================================== 23:43:35 (1746157415) [10214.800659] Lustre: lustre-MDD0000: changelog on [10214.802197] Lustre: Skipped 1 previous similar message [10224.515640] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 23:43:47 (1746157427) [10241.603339] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 23:44:04 (1746157444) [10260.900021] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 23:44:23 (1746157463) [10272.352102] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 23:44:34 (1746157474) [10283.189951] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 23:44:45 (1746157485) [10287.228711] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 23:44:49 (1746157489) [10292.430214] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 23:44:54 (1746157494) [10345.293712] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 23:45:47 (1746157547) [10352.308994] Lustre: Failing over lustre-OST0000 [10352.418244] LustreError: 30440:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10352.420752] LustreError: 30440:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10352.457278] Lustre: server umount lustre-OST0000 complete [10353.341919] LustreError: 2535:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10353.351065] LustreError: 2535:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [10354.345934] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10354.350481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10354.356557] LustreError: 2535:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10355.854738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10358.334199] LustreError: 21259:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10358.344111] LustreError: 21259:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [10361.378427] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10361.387068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10361.487041] Lustre: 31576:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [10361.590202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10361.600539] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10363.064878] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10363.540265] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10363.542010] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [10363.542011] Lustre: Skipped 3 previous similar messages [10364.095497] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [10366.110538] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 23:46:08 (1746157568) [10390.908434] Lustre: Failing over lustre-OST0000 [10390.940117] LustreError: 506:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10390.942803] LustreError: 506:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10390.962730] Lustre: server umount lustre-OST0000 complete [10391.628156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10391.628789] LustreError: 11773:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10391.628792] LustreError: 11773:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [10391.645257] Lustre: Skipped 1 previous similar message [10394.302910] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10394.310285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10394.375304] Lustre: 1150:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [10394.467363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10394.476129] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10395.573995] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10396.266735] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10396.269803] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [10396.269804] Lustre: Skipped 1 previous similar message [10396.530543] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [10398.302814] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 23:46:40 (1746157600) [10413.048746] Lustre: Failing over lustre-OST0000 [10413.084675] LustreError: 2523:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10413.089408] LustreError: 2523:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10413.113764] Lustre: server umount lustre-OST0000 complete [10413.430836] LustreError: 2529:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10413.436846] LustreError: 2529:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [10414.366055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [10414.366146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10414.374784] LustreError: Skipped 4 previous similar messages [10416.164540] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10416.172094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10416.228425] Lustre: 3125:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [10416.333931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10416.345114] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10418.075368] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10418.147402] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10418.147439] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [10418.147441] Lustre: Skipped 1 previous similar message [10418.209556] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [10419.542455] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 23:47:02 (1746157622) [10443.562833] Lustre: Failing over lustre-OST0000 [10443.594692] LustreError: 5562:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10443.605668] LustreError: 5562:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10443.636562] Lustre: server umount lustre-OST0000 complete [10446.381605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10446.382124] LustreError: 2530:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10446.382126] LustreError: 2530:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [10446.408872] Lustre: Skipped 2 previous similar messages [10447.385644] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10447.393356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10447.448039] Lustre: 6158:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [10447.551638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10447.563836] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10448.484146] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10448.976926] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10448.977147] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [10448.977148] Lustre: Skipped 1 previous similar message [10449.487018] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [10451.034862] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 23:47:33 (1746157653) [10462.799053] Lustre: Failing over lustre-OST0000 [10462.823818] LustreError: 7581:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10462.827721] LustreError: 7581:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10462.840906] Lustre: server umount lustre-OST0000 complete [10464.010165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10464.010187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10464.018981] LustreError: Skipped 8 previous similar messages [10465.868148] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10465.873815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10465.913433] Lustre: 8194:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [10465.984098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10465.990348] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10467.651286] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10467.704574] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10467.705306] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [10467.705307] Lustre: Skipped 1 previous similar message [10467.853400] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [10469.384696] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 23:47:52 (1746157672) [10475.371432] Lustre: Failing over lustre-OST0000 [10475.434834] Lustre: server umount lustre-OST0000 complete [10478.531746] LustreError: 2530:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10478.538723] LustreError: 2530:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [10482.913973] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10482.920652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10482.972245] Lustre: 10110:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [10483.064167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10483.074484] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10484.857050] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [10484.857407] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [10484.857409] Lustre: Skipped 1 previous similar message [10485.409218] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [10487.073022] Lustre: DEBUG MARKER: == sanity test 165g: ofd_access_log_reader --keepalive works ========================================================== 23:48:09 (1746157689) [10529.053600] Lustre: Failing over lustre-OST0000 [10529.085724] LustreError: 11045:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10529.118105] LustreError: 11045:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10529.143384] Lustre: server umount lustre-OST0000 complete [10529.946727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10529.953227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10529.963620] Lustre: Skipped 3 previous similar messages [10537.673169] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10537.682851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10537.737749] Lustre: 12200:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [10537.843406] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10537.852072] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10539.706359] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10539.710737] Lustre: Skipped 1 previous similar message [10539.723128] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [10539.723134] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10539.735573] Lustre: Skipped 1 previous similar message [10540.105633] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [10542.078416] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 23:49:04 (1746157744) [10542.859579] Lustre: DEBUG MARKER: creating a 10 Mb file [10579.387163] Lustre: DEBUG MARKER: starting reads [10580.187648] Lustre: DEBUG MARKER: truncating the file [10581.123432] Lustre: DEBUG MARKER: killing dd [10581.768608] Lustre: DEBUG MARKER: removing the temporary file [10584.396177] Lustre: DEBUG MARKER: == sanity test 170a: test lctl df to handle corrupted log ========================================================== 23:49:47 (1746157787) [10587.485965] Lustre: DEBUG MARKER: == sanity test 170b: check filename encoding ============= 23:49:50 (1746157790) [10598.662967] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 23:50:01 (1746157801) [10604.727800] Lustre: DEBUG MARKER: == sanity test 172: manual device removal with lctl cleanup/detach ================================================================ 23:50:07 (1746157807) [10608.797745] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 23:50:11 (1746157811) [10609.518114] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [10610.311134] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 23:50:13 (1746157813) [10611.307109] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_module obdecho/obdecho [10618.633712] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 23:50:21 (1746157821) [10619.508802] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_module obdecho/obdecho [10619.574445] Lustre: Echo OBD driver; http://www.lustre.org/ [10627.027357] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 23:50:29 (1746157829) [10675.139486] Lustre: DEBUG MARKER: == sanity test 182a: Test parallel modify metadata operations from mdc ========================================================== 23:51:17 (1746157877) [10699.110217] Lustre: DEBUG MARKER: == sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 23:51:41 (1746157901) [11093.486686] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 23:58:16 (1746158296) [11094.114032] Lustre: *** cfs_fail_loc=148, val=0*** [11097.435736] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 23:58:20 (1746158300) [11102.260002] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 23:58:24 (1746158304) [11105.911342] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 23:58:28 (1746158308) [11123.673077] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 23:58:46 (1746158326) [11128.549903] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 23:58:51 (1746158331) [11132.958929] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 23:58:55 (1746158335) [11136.377160] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 23:58:58 (1746158338) [11140.302256] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 23:59:03 (1746158343) [11147.672500] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 23:59:10 (1746158350) [11152.202546] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 23:59:14 (1746158354) [11155.205177] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 23:59:18 (1746158358) [11174.291386] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 23:59:36 (1746158376) [11177.901315] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 23:59:40 (1746158380) [11181.533257] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 23:59:44 (1746158384) [11184.870842] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 23:59:47 (1746158387) [11187.996920] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 23:59:50 (1746158390) [11192.084685] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 23:59:54 (1746158394) [11194.999890] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 23:59:57 (1746158397) [11198.145674] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 00:00:00 (1746158400) [11201.603019] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 00:00:04 (1746158404) [11205.600195] Lustre: lustre-MDD0000: changelog on [11205.603508] Lustre: Skipped 5 previous similar messages [11208.936579] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [11209.625472] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.10366 [11210.977134] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [11211.672102] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.30364 [11212.914444] Lustre: DEBUG MARKER: Test: lfs mkdir -i 1 /mnt/lustre/d205a.sanity.remote [11213.734167] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.15283 [11215.124972] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [11216.010391] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.28516 [11217.371982] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [11218.129343] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.4482 [11219.248447] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [11219.987429] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.659 [11221.157766] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11221.916865] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.6313 [11223.581374] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [11224.333995] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.6318 [11225.825527] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [11226.741024] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.21195 [11228.064367] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [11228.915820] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.3588 [11230.601881] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [11231.386404] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.22673 [11232.669538] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [11233.383283] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.6314 [11238.389429] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11239.215752] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg230-client.v [11240.565237] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11241.454485] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg230-client.E [11242.911398] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11243.801362] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.oleg230-client.v [11247.732638] Lustre: lustre-MDD0001: changelog off [11247.734572] Lustre: Skipped 6 previous similar messages [11252.561240] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 00:00:55 (1746158455) [11258.975794] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 00:01:01 (1746158461) [11262.744951] Lustre: DEBUG MARKER: == sanity test 205d: verify the format of some stats files ========================================================== 00:01:05 (1746158465) [11270.665357] Lustre: DEBUG MARKER: == sanity test 205e: verify the output of lljobstat ====== 00:01:13 (1746158473) [11281.006981] Lustre: DEBUG MARKER: == sanity test 205f: verify qos_ost_weights YAML format == 00:01:23 (1746158483) [11287.287483] Lustre: DEBUG MARKER: == sanity test 205g: stress test for job_stats procfile == 00:01:29 (1746158489) [11384.568141] Lustre: DEBUG MARKER: == sanity test 205h: check jobid xattr is stored correctly ========================================================== 00:03:06 (1746158586) [11392.265476] Lustre: DEBUG MARKER: == sanity test 205i: check job_xattr parameter accepts and rejects values correctly ========================================================== 00:03:14 (1746158594) [11400.501845] Lustre: DEBUG MARKER: == sanity test 205k: Verify '?' operator on job stats ==== 00:03:23 (1746158603) [11409.259440] Lustre: DEBUG MARKER: == sanity test 205l: Verify job stats can scale ========== 00:03:31 (1746158611) [11494.857105] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 00:04:57 (1746158697) [11499.233055] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 00:05:01 (1746158701) [11504.198370] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 00:05:06 (1746158706) [11508.618472] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 00:05:11 (1746158711) [11516.109378] Lustre: Failing over lustre-MDT0000 [11516.295851] LustreError: 21196:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11516.299002] LustreError: 21196:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11516.428260] Lustre: server umount lustre-MDT0000 complete [11518.105660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11518.111915] LustreError: Skipped 1 previous similar message [11518.115774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11518.128671] Lustre: Skipped 1 previous similar message [11518.135729] LustreError: 972:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11518.143492] LustreError: 972:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [11529.426452] LustreError: 32517:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11529.437891] LustreError: 32517:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 10 previous similar messages [11531.059809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11531.153053] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11531.326607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11531.373161] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11533.105107] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [11534.428458] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11536.323773] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [11536.329784] Lustre: Skipped 1 previous similar message [11536.356318] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11536.378479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:37068 to 0x280000403:37153) [11536.378745] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:37785 to 0x2c0000403:37825) [11538.110705] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11539.022230] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11543.659929] Lustre: Failing over lustre-MDT0000 [11543.781554] LustreError: 22489:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11543.786318] LustreError: 22489:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11543.820200] Lustre: server umount lustre-MDT0000 complete [11546.332237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11546.337595] Lustre: Skipped 3 previous similar messages [11546.338656] LustreError: 32517:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11546.338658] LustreError: 32517:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [11557.637769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11557.719292] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11557.884944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11557.913147] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11559.233339] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [11559.469772] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11562.895073] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [11562.900899] Lustre: Skipped 3 previous similar messages [11562.931378] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11562.958036] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:37785 to 0x2c0000403:37857) [11562.971006] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:37068 to 0x280000403:37185) [11564.878467] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11565.822187] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11571.899246] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 00:06:13 (1746158773) [11586.802757] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 00:06:29 (1746158789) [11592.807403] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 00:06:35 (1746158795) [11597.660798] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 00:06:40 (1746158800) [11611.610482] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 00:06:54 (1746158814) [11637.745382] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 00:07:19 (1746158839) [11642.878844] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 00:07:25 (1746158845) [11657.697986] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with embedded hyphen ('-') ========================================================== 00:07:40 (1746158860) [11664.087689] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 00:07:46 (1746158866) [11665.136056] Lustre: DEBUG MARKER: creating a 10 Mb file [11716.437202] Lustre: DEBUG MARKER: starting reads [11717.406014] Lustre: DEBUG MARKER: truncating the file [11718.637416] Lustre: DEBUG MARKER: killing dd [11719.579654] Lustre: DEBUG MARKER: removing the temporary file [11723.818896] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 00:08:46 (1746158926) [11728.701363] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 00:08:50 (1746158930) [11732.075222] Lustre: *** cfs_fail_loc=229, val=0*** [11733.372030] Lustre: *** cfs_fail_loc=229, val=0*** [11733.375579] Lustre: Skipped 1 previous similar message [11736.921182] Lustre: *** cfs_fail_loc=229, val=1*** [11739.146580] Lustre: *** cfs_fail_loc=229, val=0*** [11739.148082] Lustre: Skipped 5 previous similar messages [11749.858720] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 00:09:12 (1746158952) [11757.884839] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 00:09:20 (1746158960) [11763.548174] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 00:09:25 (1746158965) [11769.276070] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 00:09:31 (1746158971) [11774.761048] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 00:09:37 (1746158977) [11774.904698] Lustre: lustre-OST0000: Client d491cc6f-08c2-4b8d-9f5e-9fb84ce4a620 (at 192.168.202.30@tcp) reconnecting [11790.707108] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 00:09:53 (1746158993) [11793.908333] LustreError: 23480:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 sleeping for 3000ms [11796.923553] LustreError: 23480:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 awake [11804.226009] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 00:10:06 (1746159006) [11812.965050] Lustre: *** cfs_fail_loc=520, val=57344*** [11813.027990] LNet: *** cfs_fail_loc=e000, val=2147483648*** [11818.075522] Lustre: lustre-MDT0000: Client d491cc6f-08c2-4b8d-9f5e-9fb84ce4a620 (at 192.168.202.30@tcp) reconnecting [11818.105220] Lustre: 32521:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff88007a0e3b80 x1830975607668608/t34359739959(0) o35->d491cc6f-08c2-4b8d-9f5e-9fb84ce4a620@192.168.202.30@tcp:0/0 lens 392/456 e 0 to 0 dl 1746159027 ref 1 fl Interpret:/202/0 rc 0/0 job:'dd.0' uid:0 gid:0 [11829.512079] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 00:10:31 (1746159031) [11832.360171] LustreError: 23480:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 sleeping for 22000ms [11852.405399] Lustre: lustre-OST0000: Client d491cc6f-08c2-4b8d-9f5e-9fb84ce4a620 (at 192.168.202.30@tcp) reconnecting [11854.044731] LustreError: 23480:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout interrupted [11854.051600] LustreError: 23480:0:(ldlm_lib.c:3490:target_bulk_io()) @@@ bulk READ failed: rc = -107 req@ffff8800987e6300 x1830975607681408/t0(0) o3->d491cc6f-08c2-4b8d-9f5e-9fb84ce4a620@192.168.202.30@tcp:0/0 lens 488/440 e 0 to 0 dl 1746159056 ref 1 fl Interpret:/200/0 rc 0/0 job:'dd.0' uid:0 gid:0 [11854.067556] Lustre: 23480:0:(service.c:2494:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/2s); client may timeout req@ffff8800987e6300 x1830975607681408/t0(0) o3->d491cc6f-08c2-4b8d-9f5e-9fb84ce4a620@192.168.202.30@tcp:0/0 lens 488/440 e 0 to 0 dl 1746159056 ref 1 fl Complete:/200/0 rc 0/0 job:'dd.0' uid:0 gid:0 [11860.314501] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [11861.800203] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [11863.241006] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 00:11:05 (1746159065) [11868.764650] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 00:11:10 (1746159070) [11875.158095] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 00:11:17 (1746159077) [11881.295030] Lustre: DEBUG MARKER: == sanity test 226d: verify fid2path with -n and -fn option ========================================================== 00:11:23 (1746159083) [11886.512429] Lustre: DEBUG MARKER: == sanity test 226e: Verify path2fid -0 option with newline and space ========================================================== 00:11:28 (1746159088) [11891.543496] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 00:11:33 (1746159093) [11896.381125] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 00:11:38 (1746159098) [12062.762459] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 00:14:25 (1746159265) [12204.632696] Lustre: Failing over lustre-MDT0000 [12204.970151] LustreError: 4201:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [12204.972907] LustreError: 4201:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [12205.030310] Lustre: server umount lustre-MDT0000 complete [12206.862556] LustreError: 22166:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12206.878942] LustreError: 22166:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 13 previous similar messages [12209.115671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12209.124742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12209.139989] Lustre: Skipped 2 previous similar messages [12209.887283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12209.993954] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12210.223909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12210.263710] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [12211.819867] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [12212.106980] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [12215.225739] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [12215.229713] Lustre: Skipped 3 previous similar messages [12215.253359] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [12215.292628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:43398 to 0x280000403:43425) [12215.299210] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:44036 to 0x2c0000403:44065) [12247.207386] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 00:17:29 (1746159449) [12542.363046] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 00:22:24 (1746159744) [12546.892859] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 00:22:29 (1746159749) [12552.285643] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 00:22:34 (1746159754) [12594.686409] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 00:23:16 (1746159796) [12596.331695] Lustre: *** cfs_fail_loc=1801, val=0*** [12596.334003] LustreError: 11628:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240001b77:0x2f6e:0x0]/f0 failed: rc = -5 [12597.570077] LustreError: 11628:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0000: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush migrate_dir' to finish migration: rc = -1 [12597.581456] LustreError: 11628:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200004281:0x56fa:0x0]/migrate_dir failed: rc = -1 [12600.166032] Lustre: 11628:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0001: [0x240001b71:0x16:0x0]/foo is open, migrate only dentry [12606.396599] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [12607.804789] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 00:23:29 (1746159809) [12613.867716] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 00:23:36 (1746159816) [12620.817015] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 00:23:42 (1746159822) [12626.146999] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 00:23:48 (1746159828) [12630.881902] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 00:23:53 (1746159833) [12636.228437] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 00:23:58 (1746159838) [12641.909000] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 00:24:04 (1746159844) [12643.537761] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [12645.296698] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 00:24:07 (1746159847) [12725.508388] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 00:25:27 (1746159927) [12734.176362] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 00:25:36 (1746159936) [12738.907958] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 00:25:41 (1746159941) [12760.196479] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 00:26:02 (1746159962) [12789.300795] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 00:26:31 (1746159991) [12823.396940] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 00:27:05 (1746160025) [12829.355687] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 00:27:11 (1746160031) [12837.031317] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 00:27:19 (1746160039) [12843.486375] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 00:27:26 (1746160046) [12844.415138] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [12845.461750] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 00:27:27 (1746160047) [12846.633795] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [12848.112655] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 00:27:30 (1746160050) [12855.628678] Lustre: DEBUG MARKER: == sanity test 230x: dir migration check space =========== 00:27:38 (1746160058) [12865.266745] LustreError: 32531:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000007:0x1:0x0]/d230x.sanity failed: rc = -28 [12865.277618] LustreError: 32531:0:(mdt_reint.c:2523:mdt_reint_migrate()) Skipped 1 previous similar message [12901.681038] Lustre: DEBUG MARKER: == sanity test 230y: unlink dir with bad hash type ======= 00:28:24 (1746160104) [12904.632163] Lustre: *** cfs_fail_loc=1802, val=0*** [12904.747942] LustreError: 11628:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240001b77:0x35cf:0x0]/d11 failed: rc = -77 [12916.650206] Lustre: DEBUG MARKER: == sanity test 230z: resume dir migration with bad hash type ========================================================== 00:28:39 (1746160119) [12919.269302] Lustre: *** cfs_fail_loc=1802, val=0*** [12919.276383] Lustre: Skipped 1 previous similar message [12919.428725] LustreError: 32531:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240001b77:0x35d2:0x0]/d84 failed: rc = -77 [12951.553375] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 00:29:14 (1746160154) [12957.945049] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 00:29:20 (1746160160) [12999.615341] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 00:30:01 (1746160201) [13000.558262] Lustre: *** cfs_fail_loc=31c, val=0*** [13004.276233] Lustre: Failing over lustre-OST0000 [13004.438696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13004.449221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13004.461635] LustreError: 17779:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13004.461637] LustreError: 17779:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [13004.472289] Lustre: Skipped 2 previous similar messages [13004.479457] LustreError: 21258:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13004.496102] LustreError: 21258:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [13004.529224] Lustre: server umount lustre-OST0000 complete [13006.713635] LustreError: 21258:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13006.720563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13006.743323] LustreError: 21258:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [13007.986912] LustreError: 21258:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13010.105454] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13010.116608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13010.190787] Lustre: 18374:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [13010.349642] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13010.363940] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13011.427452] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13012.235404] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13012.236738] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [13012.236740] Lustre: Skipped 3 previous similar messages [13013.470794] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [13018.866690] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 00:30:21 (1746160221) [13019.975157] Lustre: *** cfs_fail_loc=31c, val=0*** [13019.977376] LustreError: 13851:0:(ldlm_request.c:475:ldlm_cli_enqueue_local()) ### delayed lvb init failed (rc -12) ns: filter-lustre-OST0000_UUID lock: ffff8800ad363a00/0xbde5d1fbbb46d79a lrc: 2/0,0 mode: --/PR res: [0x280000403:0xd6c9:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 13851 timeout: 0 lvb_type: 0 [13022.869751] Lustre: Failing over lustre-OST0000 [13022.905227] LustreError: 19421:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13022.916582] LustreError: 19421:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13022.984446] Lustre: server umount lustre-OST0000 complete [13023.391960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [13023.407496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13023.432093] LustreError: 2538:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13028.169219] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13028.179866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13028.323911] Lustre: 20018:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [13028.478286] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13028.492354] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13029.813650] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13030.426929] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13030.427138] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [13030.427139] Lustre: Skipped 1 previous similar message [13031.910473] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [13037.971662] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 00:30:40 (1746160240) [13042.663359] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 00:30:44 (1746160244) [13046.847409] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 00:30:49 (1746160249) [13052.059788] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 00:30:54 (1746160254) [13059.012859] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 00:31:01 (1746160261) [13064.239802] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 00:31:06 (1746160266) [13069.981900] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 00:31:12 (1746160272) [13148.893750] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 00:32:31 (1746160351) [13151.122422] Lustre: *** cfs_fail_loc=2100, val=0*** [13156.804903] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 00:32:39 (1746160359) [13164.364130] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 00:32:46 (1746160366) [13165.958158] Lustre: *** cfs_fail_loc=713, val=1*** [13165.961075] Lustre: Skipped 1 previous similar message [13173.486392] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 00:32:55 (1746160375) [13227.142312] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 00:33:49 (1746160429) [13250.910064] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 00:34:13 (1746160453) [13251.485199] Lustre: *** cfs_fail_loc=105, val=0*** [13254.876848] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 00:34:17 (1746160457) [13446.831413] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 00:37:29 (1746160649) [13507.609581] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 00:38:30 (1746160710) [13510.713609] Lustre: DEBUG MARKER: == sanity test 245a: check mdc connection flag/data: multiple modify RPCs ========================================================== 00:38:33 (1746160713) [13514.059340] Lustre: DEBUG MARKER: == sanity test 245b: check osp connection flag/data: multiple modify RPCs ========================================================== 00:38:36 (1746160716) [13518.323035] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 00:38:40 (1746160720) [13523.812367] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 00:38:46 (1746160726) [13527.675333] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 00:38:50 (1746160730) [13531.995555] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 00:38:54 (1746160734) [13535.821759] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 00:38:58 (1746160738) [13539.120177] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 00:39:01 (1746160741) [13539.931695] Lustre: enable_remote_subdir_mount is deprecated, it's always enabled. [13547.649777] Lustre: DEBUG MARKER: == sanity test 247g: striped directory submount revalidate ROOT from cache ========================================================== 00:39:10 (1746160750) [13553.043902] Lustre: DEBUG MARKER: == sanity test 247h: remote directory submount revalidate ROOT from cache ========================================================== 00:39:15 (1746160755) [13559.600344] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 00:39:22 (1746160762) [13616.157350] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 00:40:18 (1746160818) [13641.676431] Lustre: DEBUG MARKER: == sanity test 248c: verify whole file read behavior ===== 00:40:44 (1746160844) [13651.247750] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 00:40:54 (1746160854) [13654.496320] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 00:40:57 (1746160857) [13657.451480] Lustre: DEBUG MARKER: == sanity test 251a: Handling short read and write correctly ========================================================== 00:41:00 (1746160860) [13660.530636] Lustre: DEBUG MARKER: == sanity test 251b: short read restore offset correctly ========================================================== 00:41:03 (1746160863) [13668.225233] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 00:41:11 (1746160871) [13672.602484] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 00:41:15 (1746160875) [13707.193851] LustreError: 22166:0:(lod_qos.c:1380:lod_ost_alloc_specific()) can't lstripe objid [0x200004284:0x39:0x0]: have 0 want 1 [13745.573270] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 00:42:28 (1746160948) [13747.388841] Lustre: lustre-MDD0000: changelog on [13747.391108] Lustre: Skipped 1 previous similar message [13752.644782] Lustre: lustre-MDD0001: changelog off [13752.646272] Lustre: Skipped 1 previous similar message [13755.073298] Lustre: DEBUG MARKER: SKIP: sanity test_255a skipping excluded test 255a (base 255) [13755.799159] Lustre: DEBUG MARKER: SKIP: sanity test_255b skipping excluded test 255b (base 255) [13756.608868] Lustre: DEBUG MARKER: SKIP: sanity test_255c skipping excluded test 255c (base 255) [13757.445886] Lustre: DEBUG MARKER: SKIP: sanity test_256 skipping excluded test 256 [13758.244053] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 00:42:40 (1746160960) [13759.069173] Lustre: *** cfs_fail_loc=161, val=0*** [13759.831090] Lustre: Failing over lustre-MDT0001 [13760.601115] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.30@tcp (stopping) [13760.603948] Lustre: Skipped 5 previous similar messages [13760.731064] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13760.740648] Lustre: Skipped 3 previous similar messages [13765.587826] LustreError: 4153:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [13765.590528] LustreError: 4153:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13765.594616] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.30@tcp (stopping) [13765.598167] Lustre: Skipped 3 previous similar messages [13765.662370] Lustre: server umount lustre-MDT0001 complete [13765.737369] LustreError: 32518:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13765.746594] LustreError: 32518:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [13768.810534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13768.867441] Lustre: 4764:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [13769.034565] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13769.075260] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [13770.097346] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [13770.190230] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [13771.550312] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [13774.043855] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [13774.049017] Lustre: Skipped 1 previous similar message [13774.057388] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [13774.074472] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:23586 to 0x2c0000400:23617) [13774.074596] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:23818 to 0x280000400:23841) [13778.980194] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 00:43:01 (1746160981) [13781.400722] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 00:43:04 (1746160984) [13783.749872] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 00:43:06 (1746160986) [13796.410426] Lustre: *** cfs_fail_loc=2301, val=0*** [13797.590148] Lustre: Failing over lustre-OST0000 [13797.619782] LustreError: 7317:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13797.622298] LustreError: 7317:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [13797.701683] Lustre: server umount lustre-OST0000 complete [13797.785227] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13797.790852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13797.799276] LustreError: 20964:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13799.082758] LustreError: 20963:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13799.086070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13799.115250] LustreError: 20963:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [13801.698550] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13801.707704] LDISKFS-fs (dm-2): 1 truncate cleaned up [13801.709094] LDISKFS-fs (dm-2): recovery complete [13801.712690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13801.765602] Lustre: 7994:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [13801.860917] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13801.867905] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13803.457696] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13803.984370] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [13804.411624] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [13804.411761] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13804.419715] Lustre: Skipped 2 previous similar messages [13810.137902] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 00:43:32 (1746161012) [13813.426034] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 00:43:36 (1746161016) [13820.272194] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 00:43:42 (1746161022) [13823.520077] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 00:43:46 (1746161026) [13826.914487] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 00:43:49 (1746161029) [13830.121888] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 00:43:52 (1746161032) [13833.907918] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 00:43:56 (1746161036) [13836.471415] Lustre: Increasing provided stripe size to a minimum value 64 [13841.344231] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 00:44:04 (1746161044) [13844.051348] Lustre: *** cfs_fail_loc=168, val=40*** [13844.723569] Lustre: *** cfs_fail_loc=168, val=20*** [13844.725467] Lustre: Skipped 2 previous similar messages [13846.053651] Lustre: *** cfs_fail_loc=168, val=15*** [13846.055412] Lustre: Skipped 5 previous similar messages [13850.306368] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 00:44:13 (1746161053) [13853.485208] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 00:44:16 (1746161056) [13855.982033] Lustre: DEBUG MARKER: == sanity test 270j: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 00:44:18 (1746161058) [13859.502247] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 00:44:22 (1746161062) [13862.163128] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 00:44:24 (1746161064) [13865.295219] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 00:44:27 (1746161067) [13868.880309] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 00:44:31 (1746161071) [13927.816063] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 00:45:30 (1746161130) [13930.429865] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 00:45:33 (1746161133) [13933.186800] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 00:45:36 (1746161136) [13936.618876] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 00:45:39 (1746161139) [13939.103543] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 00:45:42 (1746161142) [13942.655262] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 00:45:45 (1746161145) [13946.035002] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 00:45:48 (1746161148) [13949.477394] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 00:45:52 (1746161152) [13952.312905] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 00:45:55 (1746161155) [13955.119682] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 00:45:58 (1746161158) [13957.300670] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 00:46:00 (1746161160) [13958.186651] LustreError: 11628:0:(mdt_io.c:571:mdt_commitrw_write()) cfs_fail_timeout id 16b sleeping for 2000ms [13960.189753] LustreError: 11628:0:(mdt_io.c:571:mdt_commitrw_write()) cfs_fail_timeout id 16b awake [13962.071160] Lustre: DEBUG MARKER: == sanity test 273c: race writeback and object destroy === 00:46:05 (1746161165) [13964.822643] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 00:46:07 (1746161167) [13968.559048] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 00:46:11 (1746161171) [13969.245686] Lustre: Failing over lustre-OST0000 [13969.249253] LustreError: 20676:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13969.286577] LustreError: 20672:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13969.289572] LustreError: 20672:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13969.305864] Lustre: server umount lustre-OST0000 complete [13969.673119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13969.676712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13969.683277] LustreError: 20965:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13969.690505] LustreError: 20965:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [13971.839140] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13971.843978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13971.881140] Lustre: 21902:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [13971.906984] LustreError: 21927:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13971.959377] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13971.966124] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13973.358982] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [13973.763425] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13973.783671] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [13973.783824] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13973.792130] Lustre: Skipped 1 previous similar message [13974.502318] Lustre: Failing over lustre-OST0000 [13974.503971] LustreError: 23210:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13974.539783] LustreError: 23205:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13974.542887] LustreError: 23205:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13974.561353] Lustre: server umount lustre-OST0000 complete [13975.934069] LustreError: 20963:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13975.941542] LustreError: 20963:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [13976.970178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13976.974889] Lustre: Skipped 2 previous similar messages [13977.147910] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13977.154173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13977.200608] Lustre: 24470:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [13977.277650] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13977.287932] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13978.601036] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13978.678943] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [13978.679078] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13978.686350] Lustre: Skipped 1 previous similar message [13978.695617] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [13979.799768] Lustre: Failing over lustre-OST0000 [13979.802334] LustreError: 25814:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13979.805003] LustreError: 25814:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [13979.842247] LustreError: 25809:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13979.844704] LustreError: 25809:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13979.859814] Lustre: server umount lustre-OST0000 complete [13982.307701] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13982.313143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13982.426772] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13982.434255] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13983.592670] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13983.875951] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [13984.026469] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [13984.026618] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13984.033603] Lustre: Skipped 1 previous similar message [13984.931422] Lustre: Failing over lustre-OST0000 [13984.931505] LustreError: 28439:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13984.984255] Lustre: server umount lustre-OST0000 complete [13985.946866] LustreError: 11773:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13985.953842] LustreError: 11773:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [13987.376261] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13987.381051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13987.435003] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13987.441277] Lustre: Skipped 2 previous similar messages [13987.448164] Lustre: 29725:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [13987.453174] Lustre: 29725:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message [13989.208706] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [13990.401686] Lustre: Failing over lustre-OST0000 [13990.441025] LustreError: 31051:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13990.443908] LustreError: 31051:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [13990.463237] Lustre: server umount lustre-OST0000 complete [13993.219370] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13993.226268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13993.302429] LustreError: 32350:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13993.305713] LustreError: 32350:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13993.356613] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13993.360700] Lustre: Skipped 1 previous similar message [13993.365209] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13993.368651] Lustre: Skipped 1 previous similar message [13994.884682] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [13994.968203] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13994.972176] Lustre: Skipped 1 previous similar message [13995.132595] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [13995.132718] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13995.132720] Lustre: Skipped 1 previous similar message [13995.145267] Lustre: Skipped 3 previous similar messages [13996.123319] Lustre: Failing over lustre-OST0000 [13996.182710] Lustre: server umount lustre-OST0000 complete [13999.116100] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13999.125797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14001.112216] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14002.424922] Lustre: Failing over lustre-OST0000 [14002.479599] Lustre: server umount lustre-OST0000 complete [14004.281849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14004.282911] LustreError: 22169:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14004.282913] LustreError: 22169:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 7 previous similar messages [14004.293322] Lustre: Skipped 6 previous similar messages [14005.334007] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14005.340865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14005.388937] Lustre: 5132:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [14005.393975] Lustre: 5132:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 2 previous similar messages [14006.992449] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14008.195756] Lustre: Failing over lustre-OST0000 [14008.232763] LustreError: 6466:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [14008.236432] LustreError: 6466:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [14008.250214] Lustre: server umount lustre-OST0000 complete [14011.511864] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14011.521529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14011.603095] LustreError: 7724:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [14011.606205] LustreError: 7724:0:(obd_class.h:479:obd_check_dev()) Skipped 4 previous similar messages [14011.688328] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [14011.693319] Lustre: Skipped 2 previous similar messages [14011.702714] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [14011.705747] Lustre: Skipped 2 previous similar messages [14012.789206] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [14012.792975] Lustre: Skipped 2 previous similar messages [14012.881595] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [14012.882626] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [14012.882628] Lustre: Skipped 5 previous similar messages [14012.892079] Lustre: Skipped 2 previous similar messages [14013.534155] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14014.912423] Lustre: Failing over lustre-OST0000 [14014.977159] Lustre: server umount lustre-OST0000 complete [14018.128500] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14018.135000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14020.268791] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14021.892099] Lustre: Failing over lustre-OST0000 [14021.945248] Lustre: server umount lustre-OST0000 complete [14025.457205] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14025.463952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14027.506839] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14029.247146] Lustre: Failing over lustre-OST0000 [14029.317331] Lustre: server umount lustre-OST0000 complete [14032.670767] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14032.676368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14035.000913] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14036.628601] Lustre: Failing over lustre-OST0000 [14036.728220] Lustre: server umount lustre-OST0000 complete [14037.867838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14037.874954] Lustre: Skipped 8 previous similar messages [14037.877416] LustreError: 2537:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14037.884633] LustreError: 2537:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 13 previous similar messages [14040.313525] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14040.320391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14040.424310] Lustre: 17839:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [14040.428780] Lustre: 17839:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 4 previous similar messages [14043.406248] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14045.050665] Lustre: Failing over lustre-OST0000 [14045.053915] LustreError: 19108:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [14045.056783] LustreError: 19108:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [14045.096181] LustreError: 19100:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [14045.103366] LustreError: 19100:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [14045.163763] Lustre: server umount lustre-OST0000 complete [14048.432974] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14048.440977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14048.596604] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [14048.600500] Lustre: Skipped 4 previous similar messages [14048.605882] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [14048.608978] Lustre: Skipped 4 previous similar messages [14049.622572] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [14049.626461] Lustre: Skipped 4 previous similar messages [14049.809522] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.130@tcp (at 0@lo) [14049.810099] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [14049.810100] Lustre: Skipped 4 previous similar messages [14049.823522] Lustre: Skipped 9 previous similar messages [14050.502902] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14051.850496] Lustre: Failing over lustre-OST0000 [14051.910276] Lustre: server umount lustre-OST0000 complete [14055.595484] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14055.600807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14057.507831] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14058.906634] Lustre: Failing over lustre-OST0000 [14058.958810] Lustre: server umount lustre-OST0000 complete [14061.835126] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14061.840479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14063.484633] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14064.670920] Lustre: Failing over lustre-OST0000 [14064.730430] Lustre: server umount lustre-OST0000 complete [14067.598340] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14067.609222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14069.685852] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14070.920386] Lustre: Failing over lustre-OST0000 [14070.973711] Lustre: server umount lustre-OST0000 complete [14073.837907] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14073.843544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14075.962337] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14077.518802] Lustre: Failing over lustre-OST0000 [14077.599733] Lustre: server umount lustre-OST0000 complete [14081.170380] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14081.178844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14083.052448] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14084.243152] Lustre: Failing over lustre-OST0000 [14084.307585] Lustre: server umount lustre-OST0000 complete [14087.261373] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14087.266614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14088.976192] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14090.243737] Lustre: Failing over lustre-OST0000 [14090.282855] Lustre: server umount lustre-OST0000 complete [14093.544283] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14093.551801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14096.187309] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14099.823666] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 00:48:22 (1746161302) [14102.924934] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 00:48:25 (1746161305) [14104.236393] Lustre: Failing over lustre-MDT0000 [14104.242001] Lustre: Failing over lustre-MDT0001 [14104.353145] LustreError: 8005:0:(tgt_mount.c:1756:server_put_super()) cfs_race id 60c sleeping [14104.573068] Lustre: server umount lustre-MDT0000 complete [14106.162886] LustreError: 30410:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14106.177809] LustreError: 30410:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 24 previous similar messages [14108.202380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14108.283656] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14108.293029] LustreError: 8586:0:(tgt_mount.c:1486:server_start_targets()) cfs_fail_race id 60c waking [14108.296890] LustreError: 8005:0:(tgt_mount.c:1756:server_put_super()) cfs_fail_race id 60c awake: rc=1059 [14108.315467] Lustre: server umount lustre-MDT0001 complete [14108.777492] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14108.786465] Lustre: Skipped 20 previous similar messages [14110.450342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14111.834213] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14115.453160] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.202.130@tcp (at 0@lo) [14115.458688] Lustre: Skipped 15 previous similar messages [14115.847363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14115.916595] Lustre: 9686:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.130@tcp: no remote llog for lustre-sptlrpc, check MGS config [14115.920690] Lustre: 9686:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 8 previous similar messages [14116.028347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [14116.032626] LustreError: Skipped 1 previous similar message [14116.123751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14116.153738] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [14116.155817] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [14116.155819] Lustre: Skipped 8 previous similar messages [14116.168260] Lustre: Skipped 8 previous similar messages [14117.498939] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14119.151771] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [14121.174716] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [14121.178290] Lustre: Skipped 7 previous similar messages [14121.199257] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:23858 to 0x280000400:23873) [14121.200063] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:23623 to 0x2c0000400:23649) [14121.232935] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:55660 to 0x2c0000403:55681) [14121.238692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:55027 to 0x280000403:55073) [14126.764231] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 00:48:49 (1746161329) [14127.565541] LustreError: 8593:0:(llog_osd.c:1480:llog_osd_open()) cfs_race id 15e sleeping [14129.249370] Lustre: Failing over lustre-MDT0000 [14129.322072] LustreError: 10730:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [14129.325463] LustreError: 10730:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [14129.337869] LustreError: 10730:0:(tgt_mount.c:128:server_deregister_mount()) cfs_fail_race id 15e waking [14129.347217] LustreError: 8593:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e awake: rc=3221 [14130.356475] LustreError: 10730:0:(tgt_mount.c:128:server_deregister_mount()) cfs_fail_race id 15e waking [14130.387595] Lustre: server umount lustre-MDT0000 complete [14134.486083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14134.580120] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14134.615340] LustreError: 11334:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [14135.710951] LustreError: 11334:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [14136.721083] LustreError: 11334:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [14138.733540] LustreError: 11334:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [14138.737456] LustreError: 11334:0:(llog_osd.c:1480:llog_osd_open()) Skipped 1 previous similar message [14142.795071] LustreError: 11333:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [14142.800513] LustreError: 11333:0:(llog_osd.c:1480:llog_osd_open()) Skipped 3 previous similar messages [14146.824837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14147.836731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:55027 to 0x280000403:55105) [14147.837651] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:55660 to 0x2c0000403:55713) [14148.568031] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug all all [14151.654018] LustreError: 11333:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [14151.657715] LustreError: 11333:0:(llog_osd.c:1480:llog_osd_open()) Skipped 5 previous similar messages [14159.537754] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 00:49:22 (1746161362) [14163.370028] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 00:49:26 (1746161366) [14187.298193] Lustre: DEBUG MARKER: == sanity test 300c: chown [14367.501250] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 00:52:49 (1746161569) [14373.230469] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ========================================================== 00:52:55 (1746161575) [14378.411978] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ========================================================== 00:53:00 (1746161580) [14383.432141] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ========================================================== 00:53:05 (1746161585) [14393.718852] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory ========================================================== 00:53:15 (1746161595) [14402.531577] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory ========================================================== 00:53:25 (1746161605) [14456.125417] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ============ 00:54:18 (1746161658) [14461.050677] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ======== 00:54:23 (1746161663) [14465.582119] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ========================================================== 00:54:28 (1746161668) [14470.498284] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS ========================================================== 00:54:33 (1746161673) [14471.273050] Lustre: DEBUG MARKER: SKIP: sanity test_300m Only for single MDT [14472.351062] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA ========================================================== 00:54:34 (1746161674) [14481.182568] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [14481.912053] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================================== 00:54:44 (1746161684) [14482.696928] Lustre: *** cfs_fail_loc=1704, val=0*** [14482.700465] Lustre: 972:0:(out_handler.c:911:out_tx_end()) lustre-MDT0001-osd: error during execution of #8 from /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:670: rc = -28 [14482.707426] LustreError: 972:0:(out_handler.c:921:out_tx_end()) lustre-MDT0001-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:425: rc = -524 [14486.200235] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ========================================================== 00:54:48 (1746161688) [14489.266868] Lustre: DEBUG MARKER: == sanity test 300r: test -1 striped directory =========== 00:54:52 (1746161692) [14493.556093] Lustre: DEBUG MARKER: == sanity test 300s: test lfs mkdir -c without -i ======== 00:54:56 (1746161696) [14497.520400] Lustre: DEBUG MARKER: == sanity test 300t: test max_mdt_stripecount ============ 00:55:00 (1746161700) [14503.691249] Lustre: DEBUG MARKER: == sanity test 300ua: basic overstriped dir sanity test == 00:55:06 (1746161706) [14508.790285] Lustre: DEBUG MARKER: == sanity test 300ub: test MDT overstriping interface [14513.525272] Lustre: DEBUG MARKER: == sanity test 300uc: test MDT overstriping as default [14517.293022] Lustre: DEBUG MARKER: == sanity test 300ud: dir split ========================== 00:55:19 (1746161719) [14609.423084] Lustre: DEBUG MARKER: == sanity test 300ue: dir merge ========================== 00:56:52 (1746161812) [14662.370571] Lustre: DEBUG MARKER: == sanity test 300uf: migrate with too many local locks == 00:57:45 (1746161865) [14666.810032] Lustre: DEBUG MARKER: == sanity test 300ug: migrate overstriped dirs =========== 00:57:49 (1746161869) [14667.565698] Lustre: 15483:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 3793 > trans_max 3200 [14667.571110] Lustre: 15483:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 6/24/0, destroy: 6/24/0 [14667.576533] Lustre: 15483:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 31/2640/0 [14667.579732] Lustre: 15483:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 0/0/0 [14667.582896] Lustre: 15483:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 47/938/0, delete: 32/50/0 [14667.604722] Lustre: 15483:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 30/30/0, ref_del: 28/28/0 [14667.610233] CPU: 2 PID: 15483 Comm: mdt_io00_005 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [14667.613866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [14667.616727] Call Trace: [14667.620188] [] dump_stack+0x19/0x1b [14667.622135] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [14667.625903] [] ? __llog_ctxt_put+0xd3/0x140 [obdclass] [14667.629020] [] top_trans_start+0x351/0xab0 [ptlrpc] [14667.631284] [] lod_trans_start+0x7f/0x2e0 [lod] [14667.633791] [] mdd_trans_start+0x14/0x20 [mdd] [14667.637636] [] mdd_migrate_object+0x12ee/0x2470 [mdd] [14667.640708] [] ? _raw_spin_unlock+0xe/0x20 [14667.642520] [] ? __mdt_stripe_get+0xe1/0x4e0 [mdt] [14667.645504] [] mdd_migrate+0x28/0x30 [mdd] [14667.651090] [] mdt_reint_migrate+0x1b9f/0x24b0 [mdt] [14667.654345] [] mdt_reint_rec+0x87/0x240 [mdt] [14667.656087] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [14667.658627] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [14667.662737] [] mdt_reint+0x67/0x150 [mdt] [14667.664528] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [14667.667400] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [14667.670804] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [14667.674061] [] ? put_prev_entity+0x31/0x400 [14667.676324] [] ? do_raw_spin_unlock+0x49/0x90 [14667.679691] [] ? _raw_spin_unlock_irq+0xe/0x30 [14667.682235] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [14667.685078] [] kthread+0xe4/0xf0 [14667.691163] [] ? kthread_create_on_node+0x140/0x140 [14667.694422] [] ret_from_fork_nospec_begin+0x7/0x21 [14667.696813] [] ? kthread_create_on_node+0x140/0x140 [14671.908983] Lustre: DEBUG MARKER: == sanity test 300uh: overstripe tunable max_stripes_per_mdt ========================================================== 00:57:54 (1746161874) [14676.258866] Lustre: DEBUG MARKER: == sanity test 300ui: overstripe is not supported on one MDT system ========================================================== 00:57:58 (1746161878) [14676.916587] Lustre: DEBUG MARKER: SKIP: sanity test_300ui 1 MDT only [14677.683059] Lustre: DEBUG MARKER: == sanity test 300uj: overstriped dir with -C -N sanity test ========================================================== 00:58:00 (1746161880) [14681.317355] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ============= 00:58:04 (1746161884) [14683.890236] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open ========================================================== 00:58:06 (1746161886) [14686.754223] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ========================================================== 00:58:09 (1746161889) [14687.390915] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [14688.173372] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ========================================================== 00:58:10 (1746161890) [14713.162489] Lustre: DEBUG MARKER: == sanity test 312: make sure ZFS adjusts its block size by write pattern ========================================================== 00:58:35 (1746161915) [14713.890457] Lustre: DEBUG MARKER: SKIP: sanity test_312 the test only applies to zfs [14714.743406] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ========================================================== 00:58:37 (1746161917) [14715.289254] Lustre: *** cfs_fail_loc=720, val=0*** [14715.292587] LustreError: 26215:0:(osd_handler.c:2213:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [14715.296804] LustreError: 26215:0:(ofd_objects.c:1014:ofd_object_punch()) lustre-OST0000: failed to stop transaction: rc = -5 [14718.405691] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ========================================================== 00:58:41 (1746161921) [14719.624290] Lustre: *** cfs_fail_loc=720, val=0*** [14719.624556] LustreError: 21259:0:(osd_handler.c:2213:osd_trans_stop()) lustre-OST0001: failed in transaction hook: rc = -5 [14719.624572] LustreError: 21259:0:(ofd_objects.c:1081:ofd_destroy()) lustre-OST0001 failed to stop transaction: -5 [14719.624652] LustreError: 21259:0:(ofd_dev.c:1765:ofd_destroy_hdl()) lustre-OST0001: error destroying object [0x2c0000403:0xded3:0x0]: -5 [14719.637482] Lustre: Skipped 1 previous similar message [14723.827352] LustreError: 11344:0:(osp_sync.c:1155:osp_sync_process_committed()) lustre-OST0000-osc-MDT0000: can't cancel record: rc = -2 [14728.830491] LustreError: 11346:0:(osp_sync.c:1155:osp_sync_process_committed()) lustre-OST0001-osc-MDT0000: can't cancel record: rc = -2 [14728.834561] LustreError: 11346:0:(osp_sync.c:1155:osp_sync_process_committed()) Skipped 1 previous similar message [14734.005933] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ============= 00:58:56 (1746161936) [14741.405205] Lustre: DEBUG MARKER: == sanity test 316: lfs migrate of file with large_xattr enabled ========================================================== 00:59:04 (1746161944) [14745.147016] Lustre: DEBUG MARKER: == sanity test 317: Verify blocks get correctly update after truncate ========================================================== 00:59:07 (1746161947) [14748.835947] Lustre: DEBUG MARKER: == sanity test 318: Verify async readahead tunables ====== 00:59:11 (1746161951) [14751.778839] Lustre: DEBUG MARKER: == sanity test 319: lost lease lock on migrate error ===== 00:59:14 (1746161954) [14757.036928] Lustre: 15483:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0000: [0x200005224:0x142:0x0]/xxx is open, migrate only dentry [14760.167855] Lustre: DEBUG MARKER: == sanity test 350: force NID mismatch path to be exercised ========================================================== 00:59:22 (1746161962) [14760.936014] LNet: *** cfs_fail_loc=e001, val=100*** [14760.941217] LNet: 24433:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-0@lo matched 1830975550368640 with NID mismatch 0@lo accepted (same peer ffff8801370e4600) [14761.437253] LNet: *** cfs_fail_loc=e001, val=100*** [14761.439156] LNet: Skipped 40 previous similar messages [14761.443010] LNet: 32523:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-0@lo matched 1830975550374016 with NID mismatch 0@lo accepted (same peer ffff8801370e4600) [14761.450429] LNet: 32523:0:(lib-ptl.c:160:lnet_try_match_md()) Skipped 42 previous similar messages [14910.428491] Lustre: DEBUG MARKER: == sanity test 360: ldiskfs unlink in a separate thread == 01:01:53 (1746162113) [14926.343494] Lustre: DEBUG MARKER: == sanity test 398a: direct IO should cancel lock otherwise lockless ========================================================== 01:02:09 (1746162129) [14930.019649] Lustre: DEBUG MARKER: == sanity test 398b: DIO and buffer IO race ============== 01:02:12 (1746162132) [15006.771292] Lustre: DEBUG MARKER: == sanity test 398c: run fio to test AIO ================= 01:03:29 (1746162209) [15039.644856] Lustre: DEBUG MARKER: == sanity test 398d: run aiocp to verify block size > stripe size ========================================================== 01:04:02 (1746162242) [15055.641963] Lustre: DEBUG MARKER: == sanity test 398e: O_Direct open cleared by fcntl doesn't cause hang ========================================================== 01:04:18 (1746162258) [15059.484107] Lustre: DEBUG MARKER: == sanity test 398f: verify aio handles ll_direct_rw_pages errors correctly ========================================================== 01:04:22 (1746162262) [15067.689496] Lustre: DEBUG MARKER: == sanity test 398g: verify parallel dio async RPC submission ========================================================== 01:04:30 (1746162270) [15068.746959] LustreError: 31205:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [15068.752779] LustreError: 31205:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 5 previous similar messages [15070.751066] LustreError: 26214:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [15070.757090] LustreError: 26214:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 3 previous similar messages [15071.004802] LustreError: 29454:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [15073.010777] LustreError: 29454:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [15073.014290] LustreError: 29454:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 4 previous similar messages [15073.046485] LustreError: 29454:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [15075.050638] LustreError: 29454:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [15075.098292] LustreError: 29454:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [15077.105240] LustreError: 29454:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [15079.182976] LustreError: 29454:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [15079.186490] LustreError: 29454:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [15081.196748] LustreError: 29454:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [15081.199888] LustreError: 29454:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [15090.591891] Lustre: DEBUG MARKER: == sanity test 398h: verify correctness of read [15101.314207] Lustre: DEBUG MARKER: == sanity test 398i: verify parallel dio handles ll_direct_rw_pages errors correctly ========================================================== 01:05:03 (1746162303) [15106.207328] Lustre: DEBUG MARKER: == sanity test 398j: test parallel dio where stripe size > rpc_size ========================================================== 01:05:08 (1746162308) [15120.006978] Lustre: DEBUG MARKER: == sanity test 398k: test enospc on first stripe ========= 01:05:22 (1746162322) [15134.750876] Lustre: DEBUG MARKER: SKIP: sanity test_398k 7196004 > 600000 skipping out-of-space test on OST0 [15135.858689] Lustre: DEBUG MARKER: == sanity test 398l: test enospc on intermediate stripe/RPC ========================================================== 01:05:38 (1746162338) [15141.631766] Lustre: DEBUG MARKER: SKIP: sanity test_398l 7176092 > 600000 skipping out-of-space test on OST0 [15157.159843] Lustre: DEBUG MARKER: == sanity test 398m: test RPC failures with parallel dio ========================================================== 01:05:59 (1746162359) [15157.733805] Lustre: *** cfs_fail_loc=20e, val=1*** [15157.735966] Lustre: Skipped 1 previous similar message [15158.757847] Lustre: *** cfs_fail_loc=20e, val=1*** [15158.761369] Lustre: Skipped 4 previous similar messages [15160.773471] Lustre: *** cfs_fail_loc=20e, val=1*** [15160.775781] Lustre: Skipped 5 previous similar messages [15163.806369] Lustre: *** cfs_fail_loc=20e, val=1*** [15163.810212] Lustre: Skipped 4 previous similar messages [15167.820301] Lustre: *** cfs_fail_loc=20e, val=1*** [15167.825062] Lustre: Skipped 2 previous similar messages [15178.861445] Lustre: *** cfs_fail_loc=20e, val=1*** [15178.864765] Lustre: Skipped 7 previous similar messages [15202.919918] Lustre: *** cfs_fail_loc=20e, val=1*** [15202.922754] Lustre: Skipped 12 previous similar messages [15235.768365] Lustre: *** cfs_fail_loc=20f, val=1*** [15235.771514] Lustre: Skipped 32 previous similar messages [15306.877821] Lustre: *** cfs_fail_loc=20e, val=2*** [15306.888568] Lustre: Skipped 56 previous similar messages [15387.658502] Lustre: DEBUG MARKER: == sanity test 398n: test append with parallel DIO ======= 01:09:50 (1746162590) [15397.740157] Lustre: DEBUG MARKER: == sanity test 398o: right kms with DIO ================== 01:10:00 (1746162600) [15398.647018] Lustre: DEBUG MARKER: sanity test_398o: @@@@@@ FAIL: bad KMS [15403.719819] Lustre: DEBUG MARKER: == sanity test 398p: race aio with buffered i/o ========== 01:10:06 (1746162606) [15436.443886] Lustre: DEBUG MARKER: == sanity test 398q: race dio with buffered i/o ========== 01:10:38 (1746162638) [15512.433923] Lustre: DEBUG MARKER: == sanity test 398r: i/o error on file read ============== 01:11:55 (1746162715) [15513.022120] Lustre: *** cfs_fail_loc=20f, val=0*** [15513.024557] Lustre: Skipped 56 previous similar messages [15571.932801] Lustre: DEBUG MARKER: == sanity test 398s: i/o error on mirror file read ======= 01:12:54 (1746162774) [15572.951007] Lustre: DEBUG MARKER: sanity test_398s: @@@@@@ FAIL: mirror resync failed