[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 311939067 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.424076] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.427142] pid_max: default: 32768 minimum: 301 [ 0.428298] Security Framework initialized [ 0.430513] SELinux: Initializing. [ 0.431785] SELinux: Starting in permissive mode [ 0.433205] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.437198] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.439121] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.441182] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.444147] Initializing cgroup subsys memory [ 0.445693] Initializing cgroup subsys devices [ 0.447079] Initializing cgroup subsys freezer [ 0.449005] Initializing cgroup subsys net_cls [ 0.450789] Initializing cgroup subsys blkio [ 0.452070] Initializing cgroup subsys perf_event [ 0.453666] Initializing cgroup subsys hugetlb [ 0.455431] Initializing cgroup subsys pids [ 0.457285] Initializing cgroup subsys net_prio [ 0.458682] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.462761] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.464350] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.465976] tlb_flushall_shift: 6 [ 0.467114] FEATURE SPEC_CTRL Present [ 0.468291] FEATURE IBPB_SUPPORT Present [ 0.470299] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.472173] Spectre V2 : Vulnerable [ 0.473210] Speculative Store Bypass: Vulnerable [ 0.475379] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.483424] ACPI: Core revision 20130517 [ 0.486472] ACPI: All ACPI Tables successfully acquired [ 0.488803] ftrace: allocating 30294 entries in 119 pages [ 0.539806] Enabling x2apic [ 0.540865] Enabled x2apic [ 0.542427] Switched APIC routing to physical x2apic. [ 0.546060] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.547911] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.551483] TSC deadline timer enabled [ 0.551624] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.553529] ... version: 2 [ 0.554864] ... bit width: 48 [ 0.556191] ... generic registers: 4 [ 0.557626] ... value mask: 0000ffffffffffff [ 0.559080] ... max period: 00007fffffffffff [ 0.560104] ... fixed-purpose events: 3 [ 0.561137] ... event mask: 000000070000000f [ 0.562459] KVM setup paravirtual spinlock [ 0.567241] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.571471] KVM setup async PF for cpu 1 [ 0.572798] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.575290] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.578546] KVM setup async PF for cpu 2 [ 0.565576] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.579079] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.581197] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.582798] Brought up 4 CPUs [ 0.582804] KVM setup async PF for cpu 3 [ 0.582809] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.586560] smpboot: Max logical packages: 1 [ 0.587927] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.592125] devtmpfs: initialized [ 0.593428] x86/mm: Memory block size: 128MB [ 0.597266] EVM: security.selinux [ 0.598376] EVM: security.ima [ 0.599563] EVM: security.capability [ 0.603888] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.606471] NET: Registered protocol family 16 [ 0.608286] cpuidle: using governor haltpoll [ 0.610401] ACPI: bus type PCI registered [ 0.612293] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.614872] PCI: Using configuration type 1 for base access [ 0.616866] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.625544] ACPI: Added _OSI(Module Device) [ 0.626658] ACPI: Added _OSI(Processor Device) [ 0.627814] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.628999] ACPI: Added _OSI(Processor Aggregator Device) [ 0.630669] ACPI: Added _OSI(Linux-Dell-Video) [ 0.632312] ACPI: EC: Look up EC in DSDT [ 0.634746] ACPI: Interpreter enabled [ 0.635980] ACPI: (supports S0 S3 S4 S5) [ 0.637110] ACPI: Using IOAPIC for interrupt routing [ 0.638780] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.641502] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.648237] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.649808] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.651993] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.654435] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.658882] acpiphp: Slot [2] registered [ 0.660406] acpiphp: Slot [3] registered [ 0.661603] acpiphp: Slot [4] registered [ 0.662571] acpiphp: Slot [5] registered [ 0.663453] acpiphp: Slot [6] registered [ 0.664722] acpiphp: Slot [7] registered [ 0.665768] acpiphp: Slot [8] registered [ 0.666948] acpiphp: Slot [9] registered [ 0.668361] acpiphp: Slot [10] registered [ 0.669969] acpiphp: Slot [11] registered [ 0.670931] acpiphp: Slot [12] registered [ 0.671946] acpiphp: Slot [13] registered [ 0.673080] acpiphp: Slot [14] registered [ 0.674429] acpiphp: Slot [15] registered [ 0.675886] acpiphp: Slot [16] registered [ 0.676684] acpiphp: Slot [17] registered [ 0.678017] acpiphp: Slot [18] registered [ 0.679458] acpiphp: Slot [19] registered [ 0.680416] acpiphp: Slot [20] registered [ 0.681819] acpiphp: Slot [21] registered [ 0.682720] acpiphp: Slot [22] registered [ 0.683802] acpiphp: Slot [23] registered [ 0.685351] acpiphp: Slot [24] registered [ 0.686665] acpiphp: Slot [25] registered [ 0.687982] acpiphp: Slot [26] registered [ 0.689319] acpiphp: Slot [27] registered [ 0.690217] acpiphp: Slot [28] registered [ 0.691022] acpiphp: Slot [29] registered [ 0.691852] acpiphp: Slot [30] registered [ 0.692960] acpiphp: Slot [31] registered [ 0.693812] PCI host bridge to bus 0000:00 [ 0.694795] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.696499] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.698177] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.700425] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.702155] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.703816] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.705487] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.706084] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.706762] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.711347] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.713394] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.715472] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.718343] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.720150] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.722806] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.723233] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.724982] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.727362] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.729244] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.737691] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.742289] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.743437] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.747792] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.764372] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.785795] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.797586] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.803150] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.812231] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.839937] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.848882] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.856219] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.862248] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.879771] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.888409] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.894628] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.900088] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.918144] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.923590] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.928317] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.931451] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.944548] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.950930] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.954168] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.957354] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.971485] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.977245] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.978996] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.980849] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.983060] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.985030] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.989103] vgaarb: loaded [ 0.990110] SCSI subsystem initialized [ 0.991459] ACPI: bus type USB registered [ 0.992833] usbcore: registered new interface driver usbfs [ 0.994707] usbcore: registered new interface driver hub [ 0.996322] usbcore: registered new device driver usb [ 0.998362] PCI: Using ACPI for IRQ routing [ 0.999576] PCI: pci_cache_line_size set to 64 bytes [ 0.999956] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.999960] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 0.999962] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.000294] NetLabel: Initializing [ 1.001321] NetLabel: domain hash size = 128 [ 1.002341] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.003940] NetLabel: unlabeled traffic allowed by default [ 1.005774] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.007144] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.013948] amd_nb: Cannot enumerate AMD northbridges [ 1.015358] Switched to clocksource kvm-clock [ 1.030318] pnp: PnP ACPI init [ 1.031037] ACPI: bus type PNP registered [ 1.032067] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.032141] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.032185] pnp 00:02: [dma 2] [ 1.032220] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.032300] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.032391] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.032448] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.032706] pnp: PnP ACPI: found 6 devices [ 1.033483] ACPI: bus type PNP unregistered [ 1.044519] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.044524] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.044526] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.044527] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.044529] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.044592] NET: Registered protocol family 2 [ 1.045955] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.048282] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.050506] TCP: Hash tables configured (established 32768 bind 32768) [ 1.051877] TCP: reno registered [ 1.052671] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.053734] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.055911] NET: Registered protocol family 1 [ 1.057914] RPC: Registered named UNIX socket transport module. [ 1.059729] RPC: Registered udp transport module. [ 1.060713] RPC: Registered tcp transport module. [ 1.061632] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.062973] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.065289] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.067290] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.069271] PCI: CLS 0 bytes, default 64 [ 1.069461] Unpacking initramfs... [ 2.376846] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.380597] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.382098] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.384006] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.386105] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.387916] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.389713] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.393273] cryptomgr_test (50) used greatest stack depth: 14480 bytes left [ 2.393748] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.393783] Initialise system trusted keyring [ 2.426900] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.429116] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.435645] zpool: loaded [ 2.436592] zbud: loaded [ 2.437870] VFS: Disk quotas dquot_6.6.0 [ 2.439278] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.442253] NFS: Registering the id_resolver key type [ 2.443960] Key type id_resolver registered [ 2.445362] Key type id_legacy registered [ 2.446724] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.449532] Key type big_key registered [ 2.450873] SELinux: Registering netfilter hooks [ 2.456268] cryptomgr_test (56) used greatest stack depth: 13968 bytes left [ 2.459557] cryptomgr_test (58) used greatest stack depth: 13664 bytes left [ 2.460142] NET: Registered protocol family 38 [ 2.460153] Key type asymmetric registered [ 2.460156] Asymmetric key parser 'x509' registered [ 2.460278] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.460419] io scheduler noop registered [ 2.460427] io scheduler deadline registered (default) [ 2.460494] io scheduler cfq registered [ 2.460499] io scheduler mq-deadline registered [ 2.460504] io scheduler kyber registered [ 2.462853] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.462909] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.479334] intel_idle: does not run on family 6 model 62 [ 2.479500] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.481777] ACPI: Power Button [PWRF] [ 2.483276] GHES: HEST is not enabled! [ 2.545739] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.609049] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.719949] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.779752] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 2.894579] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.922475] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 2.950909] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 2.954857] Non-volatile memory driver v1.3 [ 2.958250] Linux agpgart interface v0.103 [ 2.959970] crash memory driver: version 1.1 [ 2.961830] nbd: registered device at major 43 [ 2.966769] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 2.966905] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 2.966947] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 2.966981] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 2.967018] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 2.972522] virtio_blk virtio1: [vda] 53424 512-byte logical blocks (27.3 MB/26.0 MiB) [ 2.982323] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 2.982410] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 2.982446] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 2.982479] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 2.982514] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 2.988585] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.994266] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 2.994312] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 2.994386] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 2.994427] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 2.994471] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 3.000105] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.010425] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 3.010476] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 3.010521] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 3.010555] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 3.010593] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 3.017154] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.023736] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 3.023785] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 3.023828] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 3.023881] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 3.023928] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 3.029404] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.034210] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 3.034238] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 3.034263] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 3.034291] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 3.034319] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 3.040343] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.047434] rdac: device handler registered [ 3.049330] hp_sw: device handler registered [ 3.050620] emc: device handler registered [ 3.051904] libphy: Fixed MDIO Bus: probed [ 3.054865] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.056029] ehci-pci: EHCI PCI platform driver [ 3.056809] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.058498] ohci-pci: OHCI PCI platform driver [ 3.059438] uhci_hcd: USB Universal Host Controller Interface driver [ 3.060742] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.063386] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.064732] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.066198] mousedev: PS/2 mouse device common for all mice [ 3.068618] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.072283] rtc_cmos 00:05: RTC can wake from S4 [ 3.074453] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.076768] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.081651] hidraw: raw HID events driver (C) Jiri Kosina [ 3.083903] usbcore: registered new interface driver usbhid [ 3.085875] usbhid: USB HID core driver [ 3.087318] drop_monitor: Initializing network drop monitor service [ 3.089298] Netfilter messages via NETLINK v0.30. [ 3.090737] TCP: cubic registered [ 3.091692] Initializing XFRM netlink socket [ 3.093440] NET: Registered protocol family 10 [ 3.095297] NET: Registered protocol family 17 [ 3.096715] Key type dns_resolver registered [ 3.098475] mce: Using 10 MCE banks [ 3.100038] Loading compiled-in X.509 certificates [ 3.102755] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.105660] registered taskstats version 1 [ 3.110679] modprobe (70) used greatest stack depth: 13456 bytes left [ 3.116061] Key type trusted registered [ 3.122031] Key type encrypted registered [ 3.122942] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.126464] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.130488] rtc_cmos 00:05: setting system clock to 2025-01-16 01:21:23 UTC (1736990483) [ 3.134465] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.136790] Write protecting the kernel read-only data: 12288k [ 3.138952] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.141510] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.151912] random: systemd: uninitialized urandom read (16 bytes read) [ 3.156275] random: systemd: uninitialized urandom read (16 bytes read) [ 3.158542] random: systemd: uninitialized urandom read (16 bytes read) [ 3.163721] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.170507] systemd[1]: Detected virtualization kvm. [ 3.171892] systemd[1]: Detected architecture x86-64. [ 3.173344] systemd[1]: Running in initial RAM disk. [ 3.177979] systemd[1]: No hostname configured. [ 3.179480] systemd[1]: Set hostname to . [ 3.181025] random: systemd: uninitialized urandom read (16 bytes read) [ 3.183124] systemd[1]: Initializing machine ID from random generator. [ 3.197839] mkdir (88) used greatest stack depth: 13376 bytes left [ 3.233604] dracut-rootfs-g (85) used greatest stack depth: 13264 bytes left [ 3.242822] random: systemd: uninitialized urandom read (16 bytes read) [ 3.245325] random: systemd: uninitialized urandom read (16 bytes read) [ 3.248881] random: systemd: uninitialized urandom read (16 bytes read) [ 3.251644] random: systemd: uninitialized urandom read (16 bytes read) [ 3.260059] random: systemd: uninitialized urandom read (16 bytes read) [ 3.263905] random: systemd: uninitialized urandom read (16 bytes read) [ 3.279084] systemd[1]: Reached target Swap. [ 3.283776] systemd[1]: Reached target Timers. [ 3.287821] systemd[1]: Reached target Local File Systems. [ 3.293853] systemd[1]: Created slice Root Slice. [ 3.297769] systemd[1]: Created slice System Slice. [ 3.302566] systemd[1]: Listening on Journal Socket. [ 3.309118] systemd[1]: Starting Load Kernel Modules... [ 3.313840] systemd[1]: Starting dracut cmdline hook... [ 3.319029] systemd[1]: Starting Journal Service... [ 3.323782] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.331254] systemd[1]: Starting Setup Virtual Console... [ 3.334313] systemd[1]: Listening on udev Kernel Socket. [ 3.337394] systemd[1]: Reached target Slices. [ 3.341834] systemd[1]: Listening on udev Control Socket. [ 3.345557] systemd[1]: Reached target Sockets. [ 3.350193] systemd[1]: Started Load Kernel Modules. [ 3.357924] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 3.366124] systemd[1]: Started Setup Virtual Console. [ 3.370160] systemd[1]: Started Journal Service. [ 3.392455] random: fast init done [ 3.394478] tsc: Refined TSC clocksource calibration: 2399.984 MHz [ 3.868867] libata version 3.00 loaded. [ 3.872856] ata_piix 0000:00:01.1: version 2.13 [ 3.877602] scsi host0: ata_piix [ 3.881204] scsi host1: ata_piix [ 3.893502] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.895881] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 3.930181] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 3.975712] ip (316) used greatest stack depth: 13080 bytes left [ 4.019489] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 4.021650] ip (343) used greatest stack depth: 12464 bytes left [ 4.056456] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 6.187233] ip (381) used greatest stack depth: 12432 bytes left [ 7.112319] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 7.551225] systemd-journald[103]: Received SIGTERM from PID 1 (systemd). [ 7.802930] SELinux: Disabled at runtime. [ 7.805843] SELinux: Unregistering netfilter hooks [ 7.881212] ip_tables: (C) 2000-2006 Netfilter Core Team [ 7.884754] systemd[1]: Inserted module 'ip_tables' [ 8.481279] systemd-journald[567]: Received request to flush runtime journal from PID 1 [ 8.665576] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 8.734555] cryptd: max_cpu_qlen set to 1000 [ 8.741023] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 8.770654] AVX version of gcm_enc/dec engaged. [ 8.772013] AES CTR mode by8 optimization enabled [ 8.826712] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 8.850879] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 8.852740] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 8.854033] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 8.992501] EDAC MC: Ver: 3.0.0 [ 9.005939] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 9.005955] EDAC sbridge: Ver: 1.1.2 [ 11.922909] mount.nfs (772) used greatest stack depth: 10704 bytes left [ 19.051699] device-mapper: uevent: version 1.0.3 [ 19.054045] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 23.676794] libcfs: loading out-of-tree module taints kernel. [ 23.678410] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 23.687366] alg: No test for adler32 (adler32-zlib) [ 24.462937] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 29.387622] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 29.605429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 29.747291] Lustre: Lustre: Build Version: 2.15.6_6_ga1ea412 [ 29.911094] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 29.913383] LNet: Accept secure, port 988 [ 31.461536] Key type lgssc registered [ 31.766656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 34.689448] icp: module license 'CDDL' taints kernel. [ 34.691771] Disabling lock debugging due to kernel taint [ 36.283125] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 39.304405] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 43.637428] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 45.971058] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 45.973871] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 48.325465] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 48.329994] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 51.402834] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 54.520370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 54.542320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 55.619054] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 55.623110] random: crng init done [ 55.631343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 55.666320] Lustre: lustre-MDT0000: new disk, initializing [ 55.686018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 55.691528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 55.719939] mount.lustre (6811) used greatest stack depth: 9728 bytes left [ 56.604411] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 60.734493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 60.755021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 60.777903] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 60.784553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 60.787488] Lustre: Skipped 1 previous similar message [ 60.822671] Lustre: lustre-MDT0001: new disk, initializing [ 60.836305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 60.844295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 60.846542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 61.760774] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 65.944382] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 65.948127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 65.960697] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 65.963497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 65.965676] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 66.038579] Lustre: lustre-OST0000: new disk, initializing [ 66.040001] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 66.051443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 66.898213] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 69.554800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 69.556984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 71.093543] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 71.097797] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 71.116308] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 71.120024] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 71.123061] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 71.153455] Lustre: lustre-OST0001: new disk, initializing [ 71.155587] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 71.169133] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 72.096653] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 74.966754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 74.970735] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 76.985229] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 81.185130] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 87.106387] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing check_logdir /tmp/testlogs/ [ 88.174390] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing yml_node [ 89.210852] Lustre: DEBUG MARKER: Client: 2.15.6.6 [ 89.892958] Lustre: DEBUG MARKER: MDS: 2.15.6.6 [ 90.580258] Lustre: DEBUG MARKER: OSS: 2.15.6.6 [ 91.012846] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Wed Jan 15 20:22:50 EST 2025 [ 92.736559] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42b 42c 407 118c 118d [ 93.130965] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 115 135 136 230d 300o [ 94.550095] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing check_config_client /mnt/lustre [ 99.739842] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 100.624835] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 101.407927] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 103.589030] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 20:23:02 (1736990582) [ 104.579548] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 105.062273] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 20:23:04 (1736990584) [ 107.014704] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 20:23:06 (1736990586) [ 125.552909] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 20:23:24 (1736990604) [ 127.119786] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 20:23:26 (1736990606) [ 127.429072] Lustre: *** cfs_fail_loc=15b, val=0*** [ 127.430321] Lustre: *** cfs_fail_loc=15b, val=0*** [ 129.154020] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 20:23:28 (1736990608) [ 130.952695] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 20:23:30 (1736990610) [ 131.227735] Lustre: *** cfs_fail_loc=19a, val=0*** [ 131.958131] Lustre: *** cfs_fail_loc=19a, val=0*** [ 131.959386] Lustre: Skipped 2 previous similar messages [ 132.449548] LustreError: 6864:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 132.454611] LustreError: 6864:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 133.165543] Lustre: *** cfs_fail_loc=19a, val=0*** [ 133.166771] Lustre: Skipped 4 previous similar messages [ 133.651062] LustreError: 6864:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 133.654078] LustreError: 6864:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message [ 133.656429] LustreError: 6864:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 133.659225] LustreError: 6864:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1 previous similar message [ 134.673675] LustreError: 6864:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 134.676967] LustreError: 6864:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 135.412470] Lustre: *** cfs_fail_loc=19a, val=0*** [ 135.414240] Lustre: Skipped 8 previous similar messages [ 137.076572] LustreError: 6864:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 137.080650] LustreError: 6864:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message [ 137.085244] LustreError: 6864:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 137.089371] LustreError: 6864:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1 previous similar message [ 139.464933] Lustre: *** cfs_fail_loc=19a, val=0*** [ 139.466143] Lustre: Skipped 16 previous similar messages [ 141.413459] LustreError: 6864:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 141.417805] LustreError: 6864:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 6 previous similar messages [ 141.421115] LustreError: 6864:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 141.424370] LustreError: 6864:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 6 previous similar messages [ 147.615537] Lustre: *** cfs_fail_loc=19a, val=0*** [ 147.617640] Lustre: Skipped 32 previous similar messages [ 149.629768] LustreError: 8068:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0001-osd: fail to cancel 1 llog-records: rc = -5 [ 149.635049] LustreError: 8068:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 15 previous similar messages [ 149.637860] LustreError: 8068:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0001-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 149.643552] LustreError: 8068:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 15 previous similar messages [ 159.118685] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 20:23:58 (1736990638) [ 159.374986] Lustre: *** cfs_fail_loc=188, val=0*** [ 159.990800] Lustre: *** cfs_fail_loc=189, val=0*** [ 161.858738] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 162.249007] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 20:24:01 (1736990641) [ 163.816874] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 20:24:03 (1736990643) [ 165.355381] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 20:24:04 (1736990644) [ 227.720709] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 20:25:07 (1736990707) [ 232.325916] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 20:25:11 (1736990711) [ 234.307053] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 234.723767] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 20:25:14 (1736990714) [ 236.624597] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 20:25:16 (1736990716) [ 251.575044] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 20:25:31 (1736990731) [ 252.221657] Lustre: *** cfs_fail_loc=725, val=0*** [ 252.997597] Lustre: *** cfs_fail_loc=725, val=0*** [ 254.722533] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 20:25:34 (1736990734) [ 256.967241] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 20:25:36 (1736990736) [ 294.241596] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 20:26:13 (1736990773) [ 304.389739] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 20:26:23 (1736990783) [ 306.301437] Lustre: *** cfs_fail_loc=513, val=0*** [ 306.302680] LustreError: 10053:0:(service.c:2115:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1821366558463296 [ 306.785669] Lustre: Failing over lustre-OST0000 [ 306.797742] Lustre: server umount lustre-OST0000 complete [ 309.954739] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 309.954750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.955002] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.967575] LustreError: Skipped 1 previous similar message [ 310.461677] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.467981] LustreError: Skipped 1 previous similar message [ 315.362903] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 315.367624] LustreError: Skipped 1 previous similar message [ 319.016303] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 319.020137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 319.055261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 319.896641] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 320.468694] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 321.133665] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.137@tcp (at 0@lo) [ 321.133675] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 321.140174] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2518 to 0x0:2561 [ 322.867961] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 323.246975] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 326.406384] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 20:26:45 (1736990805) [ 327.960203] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 20:26:47 (1736990807) [ 329.576147] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 20:26:49 (1736990809) [ 331.187645] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 20:26:50 (1736990810) [ 332.838077] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 20:26:52 (1736990812) [ 334.469746] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 20:26:53 (1736990813) [ 335.983960] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 20:26:55 (1736990815) [ 337.515684] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 20:26:56 (1736990816) [ 339.083812] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 20:26:58 (1736990818) [ 340.736249] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 20:27:00 (1736990820) [ 343.218881] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 20:27:02 (1736990822) [ 343.732206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.735705] Lustre: Skipped 1 previous similar message [ 343.737207] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 343.739496] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.137@tcp (at 0@lo) [ 343.739643] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:36 to 0x280000400:65 [ 343.745943] Lustre: Skipped 1 previous similar message [ 343.956043] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2564 to 0x0:2593 [ 344.175179] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4 to 0x2c0000400:33 [ 344.393108] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 344.395496] Lustre: Skipped 2 previous similar messages [ 344.397185] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2513 to 0x0:2529 [ 347.288533] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 350.117090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.120309] Lustre: Skipped 3 previous similar messages [ 350.121716] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 350.123982] LustreError: 167-0: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 350.126914] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.137@tcp (at 0@lo) [ 350.127085] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:36 to 0x280000400:97 [ 350.133086] Lustre: Skipped 3 previous similar messages [ 350.911631] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 350.955409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 351.761942] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 351.806475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 354.704397] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 357.672163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.675941] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 357.678137] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 357.682707] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.137@tcp (at 0@lo) [ 357.682873] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3594 to 0x0:3617 [ 358.504630] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 358.552570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 359.440770] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 359.484749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 362.363048] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 365.431745] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.436142] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 365.438616] LustreError: 167-0: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 365.442713] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.203.137@tcp (at 0@lo) [ 365.442891] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:35 to 0x2c0000400:65 [ 366.316252] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 366.363147] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 367.239092] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 367.286881] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 370.286535] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 373.386407] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 373.390337] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3532 to 0x0:3553 [ 374.264772] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 374.311509] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 375.179101] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 375.226160] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 376.698452] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 20:27:36 (1736990856) [ 378.272886] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 20:27:37 (1736990857) [ 379.782472] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 20:27:39 (1736990859) [ 393.163142] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 20:27:52 (1736990872) [ 396.107069] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 20:27:55 (1736990875) [ 396.614693] Lustre: *** cfs_fail_loc=217, val=0*** [ 397.495037] Lustre: *** cfs_fail_loc=217, val=0*** [ 397.496655] Lustre: Skipped 1 previous similar message [ 399.285020] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 399.680470] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 20:27:59 (1736990879) [ 401.345110] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 20:28:00 (1736990880) [ 403.311961] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 20:28:02 (1736990882) [ 431.033336] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 20:28:30 (1736990910) [ 432.607124] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 20:28:32 (1736990912) [ 434.094670] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 20:28:33 (1736990913) [ 435.578870] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 20:28:35 (1736990915) [ 445.178891] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 20:28:44 (1736990924) [ 452.972143] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 20:28:52 (1736990932) [ 454.853336] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 20:28:54 (1736990934) [ 455.021149] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.37@tcp inode [0x200000407:0xc14:0x0] object 0x0:3883 extent [0-4194303]: client csum cbad0603, server csum cbad0602 [ 456.136331] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 457.257926] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.37@tcp inode [0x200000407:0xc14:0x0] object 0x0:3883 extent [0-4194303], client returned csum 5e133886 (type 1), server csum d918dddc (type 1) [ 457.897007] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 459.013498] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.37@tcp inode [0x200000407:0xc14:0x0] object 0x0:3883 extent [0-4194303], client returned csum 3d16ed63 (type 2), server csum f35dedd8 (type 2) [ 459.644031] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 460.694690] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.37@tcp inode [0x200000407:0xc14:0x0] object 0x0:3883 extent [0-4194303], client returned csum cf4d6c96 (type 4), server csum cbad0602 (type 4) [ 461.362047] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 463.116309] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 464.204988] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.37@tcp inode [0x200000407:0xc14:0x0] object 0x0:3883 extent [0-4194303], client returned csum eeeaeff8 (type 20), server csum 46bbef83 (type 20) [ 464.210924] LustreError: Skipped 1 previous similar message [ 464.862031] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 466.365826] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 468.047911] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 469.634752] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 20:29:09 (1736990949) [ 471.709260] Lustre: 9267:0:(tgt_handler.c:1903:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000407:0xc15:0x0]:[0-1048575]-9f0c55c9-a46bae53 [ 471.715034] LustreError: dumping log to /tmp/lustre-log.1736990952.9267 [ 471.733977] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.37@tcp inode [0x200000407:0xc15:0x0] object 0x0:3884 extent [0-1048575], client returned csum 9f0c55c9 (type 4), server csum a46bae53 (type 4) [ 471.739369] LustreError: Skipped 2 previous similar messages [ 480.721004] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 20:29:20 (1736990960) [ 480.839406] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.37@tcp inode [0x200000407:0xc17:0x0] object 0x0:3818 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 483.066554] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.37@tcp inode [0x200000407:0xc17:0x0] object 0x0:3818 extent [0-4194303], client returned csum f5a99216 (type 4), server csum de2bf73e (type 4) [ 484.640712] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 20:29:24 (1736990964) [ 485.032693] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 485.130205] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.37@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 487.792440] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.37@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 487.799017] LustreError: Skipped 4 previous similar messages [ 494.800872] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.37@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 494.807039] LustreError: Skipped 3 previous similar messages [ 505.777395] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.37@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 505.783511] LustreError: Skipped 3 previous similar messages [ 523.920617] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.37@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 523.925593] LustreError: Skipped 5 previous similar messages [ 530.371602] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 557.868552] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.37@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [0-4194303]: client csum 7d33b9a0, server csum 7d33b99f [ 557.872654] LustreError: Skipped 17 previous similar messages [ 576.321644] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 621.966812] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.37@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 621.972684] LustreError: Skipped 26 previous similar messages [ 622.419144] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 668.635036] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 714.667932] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 754.439189] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.37@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [4194304-8388607]: client csum a30822f0, server csum a30822ef [ 754.447675] LustreError: Skipped 62 previous similar messages [ 761.002327] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 807.092426] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 808.672912] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 20:34:48 (1736991288) [ 809.000256] Lustre: *** cfs_fail_loc=21a, val=0*** [ 810.867030] Lustre: *** cfs_fail_loc=21b, val=0*** [ 814.184333] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 20:34:53 (1736991293) [ 814.412049] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 814.952104] Lustre: Modifying parameter lustre.osc.lustre*.checksums in log params [ 817.641759] Lustre: Disabling parameter lustre.osc.lustre*.checksums in log params [ 819.932209] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 821.451504] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 20:35:00 (1736991300) [ 821.833478] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 822.222995] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 824.063490] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 839.685822] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in IDLE state after 15 sec [ 841.001183] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 841.342375] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in FULL state after 0 sec [ 841.700896] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 842.987743] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 864.918224] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in IDLE state after 21 sec [ 866.851982] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 867.416454] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in FULL state after 0 sec [ 868.066520] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 870.030419] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 889.982040] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in IDLE state after 19 sec [ 891.731086] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 892.204253] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in FULL state after 0 sec [ 892.796622] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 894.607823] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 914.445710] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in IDLE state after 19 sec [ 915.888037] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 916.221341] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in FULL state after 0 sec [ 916.580992] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 917.880439] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 939.627160] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in IDLE state after 21 sec [ 941.259342] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 941.608855] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in FULL state after 0 sec [ 941.990620] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 943.335600] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 965.057906] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in IDLE state after 21 sec [ 967.007775] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 967.413898] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in FULL state after 0 sec [ 968.012824] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 969.845626] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 989.864408] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in IDLE state after 19 sec [ 991.696759] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid 40 [ 992.197127] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012b386000.ost_server_uuid in FULL state after 0 sec [ 994.013784] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 994.418327] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 20:37:53 (1736991473) [ 996.024729] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 20:37:55 (1736991475) [ 996.451448] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 996.835714] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 997.323116] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 997.766369] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 999.267699] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 999.678062] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 20:37:59 (1736991479) [ 1002.757269] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 20:38:02 (1736991482) [ 1004.764432] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ================================================================================= 20:38:04 (1736991484) [ 1015.589917] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 20:38:15 (1736991495) [ 1018.348500] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 20:38:17 (1736991497) [ 1018.736744] Lustre: *** cfs_fail_loc=228, val=0*** [ 1020.331209] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 20:38:19 (1736991499) [ 1020.635699] Lustre: *** cfs_fail_loc=228, val=0*** [ 1022.570971] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 20:38:21 (1736991501) [ 1027.269606] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ===================================================================== 20:38:26 (1736991506) [ 1030.598571] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 20:38:30 (1736991510) [ 1052.160107] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 20:38:51 (1736991531) [ 1056.349617] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 20:38:55 (1736991535) [ 1066.640174] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 20:39:05 (1736991545) [ 1103.706288] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 20:39:42 (1736991582) [ 1110.899568] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 20:39:50 (1736991590) [ 1113.372031] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 20:39:52 (1736991592) [ 1121.556933] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 20:40:00 (1736991600) [ 1124.295266] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 20:40:03 (1736991603) [ 1126.653511] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 20:40:06 (1736991606) [ 1133.968457] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 20:40:13 (1736991613) [ 1135.703886] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 20:40:15 (1736991615) [ 1138.472798] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 20:40:17 (1736991617) [ 1140.523963] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 20:40:19 (1736991619) [ 1143.671871] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 20:40:23 (1736991623) [ 1148.066838] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 20:40:27 (1736991627) [ 1148.521411] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 1149.129571] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 1149.647536] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 1150.105400] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 1152.324435] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 20:40:31 (1736991631) [ 1153.283402] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 1153.916311] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 1154.485723] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 1154.919423] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 1155.338211] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 1156.974117] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 20:40:36 (1736991636) [ 1158.625047] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 20:40:37 (1736991637) [ 1161.889748] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 20:40:41 (1736991641) [ 1163.699048] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 20:40:43 (1736991643) [ 1165.318125] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 20:40:44 (1736991644) [ 1167.647301] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 20:40:46 (1736991646) [ 1169.827562] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 20:40:49 (1736991649) [ 1172.347991] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 20:40:51 (1736991651) [ 1174.835990] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 20:40:54 (1736991654) [ 1177.009565] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 20:40:56 (1736991656) [ 1178.689199] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 20:40:58 (1736991658) [ 1181.230657] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 20:41:00 (1736991660) [ 1216.080163] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 20:41:35 (1736991695) [ 1227.496665] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 20:41:46 (1736991706) [ 1229.142520] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 20:41:48 (1736991708) [ 1432.292691] Lustre: 6274:0:(osd_handler.c:1950:osd_trans_start()) lustre-MDT0000: credits 64189 > trans_max 3200 [ 1432.295832] Lustre: 6274:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 1000/4000/0, destroy: 1/4/0 [ 1432.298499] Lustre: 6274:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 1004/148/0 [ 1432.301647] Lustre: 6274:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 5001/43010/0, punch: 0/0/0, quota 0/0/0 [ 1432.305126] Lustre: 6274:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 1001/17016/0, delete: 2/5/0 [ 1432.308663] Lustre: 6274:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 1432.311686] Pid: 6274, comm: mdt00_006 3.10.0-7.9-debug #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 1432.314426] Call Trace: [ 1432.315441] [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [ 1432.317143] [<0>] libcfs_debug_dumpstack+0x26/0x30 [libcfs] [ 1432.318998] [<0>] osd_trans_start+0x5ba/0x5e0 [osd_ldiskfs] [ 1432.320807] [<0>] top_trans_start+0x763/0xa10 [ptlrpc] [ 1432.322543] [<0>] lod_trans_start+0x34/0x40 [lod] [ 1432.324146] [<0>] mdd_trans_start+0x14/0x20 [mdd] [ 1432.325685] [<0>] mdd_unlink+0x5e3/0xdb0 [mdd] [ 1432.327101] [<0>] mdt_reint_unlink+0xe32/0x1df0 [mdt] [ 1432.328718] [<0>] mdt_reint_rec+0x87/0x240 [mdt] [ 1432.330189] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [ 1432.331870] [<0>] mdt_reint+0x67/0x150 [mdt] [ 1432.333202] [<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc] [ 1432.335075] [<0>] ptlrpc_server_handle_request+0x250/0xc30 [ptlrpc] [ 1432.337147] [<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc] [ 1432.338679] [<0>] kthread+0xe4/0xf0 [ 1432.339772] [<0>] ret_from_fork_nospec_begin+0x7/0x21 [ 1432.341464] [<0>] 0xfffffffffffffffe [ 1432.342982] Lustre: 6274:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 3200 < left 43010, rollback = 7 [ 1434.027223] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 20:45:13 (1736991913) [ 1435.134071] Lustre: lustre-MDD0000: changelog on [ 1436.068131] Lustre: lustre-MDD0001: changelog on [ 1438.603630] Lustre: lustre-MDD0001: changelog off [ 1439.526842] Lustre: lustre-MDD0000: changelog off [ 1440.168076] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 20:45:19 (1736991919) [ 1440.252179] Lustre: lustre-OST0000: Client 5ac385aa-3604-4e07-92a7-1353163b8cd8 (at 192.168.203.37@tcp) reconnecting [ 1440.255132] Lustre: Skipped 1 previous similar message [ 1441.588112] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ab273000.ost_server_uuid 40 [ 1441.941846] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800ab273000.ost_server_uuid in FULL state after 0 sec [ 1444.002581] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 20:45:23 (1736991923) [ 1446.080276] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 20:45:25 (1736991925) [ 1446.591265] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [ 1447.083074] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 20:45:26 (1736991926) [ 1448.741302] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 20:45:28 (1736991928) [ 1450.281859] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 20:45:29 (1736991929) [ 1451.788332] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 20:45:31 (1736991931) [ 1463.972150] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 20:45:43 (1736991943) [ 1465.743537] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 20:45:45 (1736991945) [ 1467.348661] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 20:45:46 (1736991946) [ 1470.243499] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 20:45:49 (1736991949) [ 1471.981674] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [ 1472.396136] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 20:45:51 (1736991951) [ 1528.855048] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 20:46:48 (1736992008) [ 1529.624600] Lustre: *** cfs_fail_loc=147, val=0*** [ 1531.744846] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 20:46:51 (1736992011) [ 1533.319629] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 20:46:52 (1736992012) [ 1534.930605] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 20:46:54 (1736992014) [ 1535.465244] Lustre: *** cfs_fail_loc=217, val=0*** [ 1535.466913] Lustre: Skipped 33 previous similar messages [ 1537.260993] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 1537.664691] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 1538.068203] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 20:46:57 (1736992017) [ 1539.765244] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 20:46:59 (1736992019) [ 1541.519606] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 20:47:00 (1736992020) [ 1542.051813] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1543.064791] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1545.071625] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1547.270443] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1551.274482] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1553.167521] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 20:47:12 (1736992032) [ 1561.287113] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 20:47:20 (1736992040) [ 1561.703848] Lustre: *** cfs_fail_loc=220, val=0*** [ 1561.705441] Lustre: Skipped 3 previous similar messages [ 1563.625033] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 20:47:23 (1736992043) [ 1573.248640] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 20:47:32 (1736992052) [ 1574.941439] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 20:47:34 (1736992054) [ 1576.531735] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 20:47:35 (1736992055) [ 1580.863831] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 20:47:40 (1736992060) [ 1582.395448] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 20:47:41 (1736992061) [ 1584.023291] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 20:47:43 (1736992063) [ 1585.600809] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ========================================================== 20:47:45 (1736992065) [ 1587.003548] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [ 1590.730953] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 20:47:50 (1736992070) [ 1592.888547] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 20:47:52 (1736992072) [ 1594.975125] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 20:47:54 (1736992074) [ 1597.456490] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 20:47:56 (1736992076) [ 1599.606427] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 20:47:59 (1736992079) [ 1608.750117] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 20:48:08 (1736992088) [ 1618.687584] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 20:48:17 (1736992097) [ 1668.439379] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 20:49:07 (1736992147) [ 1670.003747] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 20:49:09 (1736992149) [ 1671.099906] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 0 sec [ 1671.611590] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 1676.519431] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 2 sec [ 1677.393014] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 0 sec [ 1720.066661] Lustre: DEBUG MARKER: ls -l 10000 files without statahead: 24 sec [ 1724.393399] Lustre: DEBUG MARKER: ls -l 10000 files with statahead: 3 sec [ 1724.912653] Lustre: DEBUG MARKER: ls -l done [ 1744.305722] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity/: 19 seconds [ 1744.869562] Lustre: DEBUG MARKER: rm done [ 1747.182179] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 20:50:26 (1736992226) [ 1747.755720] Lustre: DEBUG MARKER: SKIP: sanity test_123ab Test must be statx() syscall supported [ 1748.420245] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 20:50:27 (1736992227) [ 1749.002043] Lustre: DEBUG MARKER: SKIP: sanity test_123ac Test must be statx() syscall supported [ 1749.679642] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 20:50:28 (1736992228) [ 1760.107111] Lustre: DEBUG MARKER: ls done [ 1764.982694] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 20:50:44 (1736992244) [ 1768.161701] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 20:50:47 (1736992247) [ 1768.805177] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 1777.802014] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ab241000 [ 1778.360066] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ab241000 [ 1778.946588] Lustre: DEBUG MARKER: LRU=1004 [ 1779.359122] Lustre: DEBUG MARKER: LIMIT=46624 [ 1779.954390] Lustre: DEBUG MARKER: LVF=5572500 [ 1780.562599] Lustre: DEBUG MARKER: OLD_LVF=100 [ 1781.152579] Lustre: DEBUG MARKER: Sleep 50 sec [ 1831.866164] Lustre: DEBUG MARKER: Dropped 390 locks in 50s [ 1832.440130] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 1840.584409] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 20:51:59 (1736992319) [ 1869.335960] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 1875.622132] Lustre: DEBUG MARKER: ls -la time: 5 seconds [ 1876.206110] Lustre: DEBUG MARKER: lru_size = 400 [ 1926.103917] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 1932.102841] Lustre: DEBUG MARKER: ls -la time: 5 seconds [ 1932.481675] Lustre: DEBUG MARKER: lru_size = 4006 [ 1933.026220] Lustre: DEBUG MARKER: lru resize performs the same with no lru resize [ 1945.792135] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 20:53:45 (1736992425) [ 1968.833721] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize diasbaled ========================================================== 20:54:08 (1736992448) [ 1991.663101] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 20:54:31 (1736992471) [ 1993.150216] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 20:54:32 (1736992472) [ 1995.229896] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 20:54:34 (1736992474) [ 1997.794300] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 20:54:37 (1736992477) [ 1999.997112] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [ 2005.372543] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 20:54:44 (1736992484) [ 2007.760955] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 20:54:46 (1736992486) [ 2012.747971] Lustre: 11027:0:(osd_handler.c:586:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 1805, FID: [0x20000040b:0x23aa:0x0]) is approaching max size limit [ 2013.705052] Lustre: 6834:0:(osd_handler.c:586:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 1805, FID: [0x20000040b:0x23aa:0x0]) is approaching max size limit [ 2017.051306] Lustre: 11027:0:(osd_handler.c:582:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 1805, FID: [0x20000040b:0x23aa:0x0]) has reached max size limit [ 2027.165653] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 20:55:06 (1736992506) [ 2029.319800] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 20:55:08 (1736992508) [ 2031.851650] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 20:55:11 (1736992511) [ 2034.237275] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 20:55:13 (1736992513) [ 2034.634093] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [ 2035.018154] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 20:55:14 (1736992514) [ 2044.120176] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 20:55:23 (1736992523) [ 2046.135289] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 20:55:25 (1736992525) [ 2049.864125] Lustre: 6860:0:(osd_handler.c:1950:osd_trans_start()) lustre-MDT0000: credits 12989 > trans_max 3200 [ 2049.868279] Lustre: 6860:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 200/800/0, destroy: 1/4/0 [ 2049.872046] Lustre: 6860:0:(osd_handler.c:1879:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2049.876377] Lustre: 6860:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 204/148/0 [ 2049.880746] Lustre: 6860:0:(osd_handler.c:1886:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2049.885042] Lustre: 6860:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 1001/8610/0, punch: 0/0/0, quota 0/0/0 [ 2049.889434] Lustre: 6860:0:(osd_handler.c:1896:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2049.893405] Lustre: 6860:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 201/3416/0, delete: 2/5/0 [ 2049.897151] Lustre: 6860:0:(osd_handler.c:1903:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2049.900997] Lustre: 6860:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 2049.904457] Lustre: 6860:0:(osd_handler.c:1910:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2049.908022] Pid: 6860, comm: mdt00_003 3.10.0-7.9-debug #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 2049.910859] Call Trace: [ 2049.911902] [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [ 2049.913556] [<0>] libcfs_debug_dumpstack+0x26/0x30 [libcfs] [ 2049.915602] [<0>] osd_trans_start+0x5ba/0x5e0 [osd_ldiskfs] [ 2049.917501] [<0>] top_trans_start+0x763/0xa10 [ptlrpc] [ 2049.919223] [<0>] lod_trans_start+0x34/0x40 [lod] [ 2049.920831] [<0>] mdd_trans_start+0x14/0x20 [mdd] [ 2049.922411] [<0>] mdd_unlink+0x5e3/0xdb0 [mdd] [ 2049.923964] [<0>] mdt_reint_unlink+0xe32/0x1df0 [mdt] [ 2049.925593] [<0>] mdt_reint_rec+0x87/0x240 [mdt] [ 2049.927058] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [ 2049.928602] [<0>] mdt_reint+0x67/0x150 [mdt] [ 2049.929998] [<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc] [ 2049.931652] [<0>] ptlrpc_server_handle_request+0x250/0xc30 [ptlrpc] [ 2049.933357] [<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc] [ 2049.934775] [<0>] kthread+0xe4/0xf0 [ 2049.935633] [<0>] ret_from_fork_nospec_begin+0x7/0x21 [ 2049.937036] [<0>] 0xfffffffffffffffe [ 2049.938194] Lustre: 6860:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 3199 < left 8610, rollback = 7 [ 2049.940859] Lustre: 6860:0:(osd_internal.h:1333:osd_trans_exec_op()) Skipped 4000 previous similar messages [ 2050.601525] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 20:55:29 (1736992529) [ 2052.487219] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 20:55:31 (1736992531) [ 2054.555101] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 20:55:33 (1736992533) [ 2056.718894] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 20:55:36 (1736992536) [ 2059.165004] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 20:55:38 (1736992538) [ 2061.267145] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 20:55:40 (1736992540) [ 2066.349120] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 20:55:45 (1736992545) [ 2070.405622] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 20:55:49 (1736992549) [ 2085.528743] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 20:56:04 (1736992564) [ 2093.734867] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 20:56:13 (1736992573) [ 2097.639648] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 20:56:16 (1736992576) [ 2102.915130] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.915338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2102.925461] Lustre: Skipped 4 previous similar messages [ 2106.945144] Lustre: server umount lustre-MDT0000 complete [ 2107.923064] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.931520] LustreError: Skipped 4 previous similar messages [ 2109.637794] LustreError: 8058:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1736992590 with bad export cookie 12608937444393934757 [ 2109.638662] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2109.643300] LustreError: 8058:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2109.730450] Lustre: server umount lustre-MDT0001 complete [ 2118.406506] Lustre: 8687:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736992592/real 1736992592] req@ffff880084f9ed40 x1821366568773504/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1736992598 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2118.432603] Lustre: server umount lustre-OST0000 complete [ 2119.930472] Lustre: 3324:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736992593/real 1736992593] req@ffff88008320d0c0 x1821366568773632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1736992600 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 2119.943493] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.017503] Lustre: server umount lustre-OST0001 complete [ 2123.238813] device-mapper: core: cleaned up [ 2126.193103] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2126.809985] Key type lgssc unregistered [ 2126.891654] LNet: 10233:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.894856] LNet: Removed LNI 192.168.203.137@tcp [ 2132.109019] alg: No test for adler32 (adler32-zlib) [ 2132.886405] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2132.983616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2133.060776] Lustre: Lustre: Build Version: 2.15.6_6_ga1ea412 [ 2133.101594] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2133.103200] LNet: Accept secure, port 988 [ 2134.625543] Key type lgssc registered [ 2134.955785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2139.592009] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2140.941090] device-mapper: uevent: version 1.0.3 [ 2140.943762] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 2142.872092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.018631] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.033937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.304953] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2149.032481] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.313467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.386338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2150.308701] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2153.870508] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2153.873578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2153.942390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.942722] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.944743] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11136 to 0x280000400:11169 [ 2155.096517] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2158.942403] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2158.949932] LustreError: Skipped 1 previous similar message [ 2158.953521] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16651 to 0x0:16673 [ 2159.054229] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2159.060632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2159.123692] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2159.167253] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:10959 to 0x2c0000400:10977 [ 2159.173624] Lustre: lustre-OST0001: deleting orphan objects from 0x0:16490 to 0x0:16513 [ 2159.962896] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2163.661179] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2164.504031] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2171.786760] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 20:57:31 (1736992651) [ 2189.170895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.171234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.178981] Lustre: Skipped 2 previous similar messages [ 2192.319567] Lustre: server umount lustre-MDT0000 complete [ 2194.179020] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.186221] LustreError: Skipped 3 previous similar messages [ 2195.018822] LustreError: 19867:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1736992675 with bad export cookie 15268994797597942873 [ 2195.020056] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2195.024073] LustreError: 19867:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2195.113671] Lustre: server umount lustre-MDT0001 complete [ 2203.731428] Lustre: 21058:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736992678/real 1736992678] req@ffff88008a792ac0 x1821368766231424/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1736992684 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2203.746612] Lustre: server umount lustre-OST0000 complete [ 2206.188524] Lustre: 10753:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736992679/real 1736992679] req@ffff8801372fcc00 x1821368766231552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1736992686 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 2206.201550] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.208306] Lustre: Skipped 1 previous similar message [ 2206.479391] Lustre: server umount lustre-OST0001 complete [ 2208.386667] device-mapper: core: cleaned up [ 2211.286135] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2212.067728] Key type lgssc unregistered [ 2212.147896] LNet: 22587:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.152669] LNet: Removed LNI 192.168.203.137@tcp [ 2216.474679] alg: No test for adler32 (adler32-zlib) [ 2217.246957] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2217.375925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2217.423609] Lustre: Lustre: Build Version: 2.15.6_6_ga1ea412 [ 2217.468110] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2217.471005] LNet: Accept secure, port 988 [ 2219.002556] Key type lgssc registered [ 2219.321990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2223.303148] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2224.760955] device-mapper: uevent: version 1.0.3 [ 2224.762677] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 2226.917201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.052766] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.067390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.074736] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2232.449196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.560451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2233.620861] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2237.346083] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2237.352715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2237.462030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2237.480815] mount.lustre (27236) used greatest stack depth: 9568 bytes left [ 2238.460748] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.471926] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11136 to 0x280000400:11201 [ 2238.698336] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2241.900021] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2241.905770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2241.968640] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2242.967937] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16651 to 0x0:16705 [ 2242.968527] Lustre: lustre-OST0001: deleting orphan objects from 0x0:16490 to 0x0:16545 [ 2242.976130] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:10959 to 0x2c0000400:11009 [ 2243.029031] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2246.992326] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2248.202001] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2255.684980] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 20:58:54 (1736992734) [ 2450.248836] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 21:02:09 (1736992929) [ 2453.979491] Lustre: *** cfs_fail_loc=327, val=0*** [ 2468.211735] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 21:02:27 (1736992947) [ 2469.881018] Lustre: *** cfs_fail_loc=328, val=0*** [ 2469.882602] Lustre: Skipped 512 previous similar messages [ 2470.888846] Lustre: *** cfs_fail_loc=328, val=0*** [ 2473.394815] Lustre: *** cfs_fail_loc=328, val=0*** [ 2473.396222] Lustre: Skipped 252 previous similar messages [ 2477.663078] Lustre: *** cfs_fail_loc=328, val=0*** [ 2477.664775] Lustre: Skipped 125 previous similar messages [ 2485.815740] Lustre: *** cfs_fail_loc=328, val=0*** [ 2485.817060] Lustre: Skipped 65 previous similar messages [ 2490.066277] LustreError: 17158:0:(ldlm_resource.c:133:seq_watermark_write()) Failed to set lock_reclaim_threshold_mb, rc = -22. [ 2493.775713] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [ 2494.212947] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [ 2494.668853] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 21:02:54 (1736992974) [ 2501.421398] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 21:03:00 (1736992980) [ 2511.122321] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 21:03:10 (1736992990) [ 2521.230730] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 21:03:20 (1736993000) [ 2531.665066] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 21:03:30 (1736993010) [ 2541.687236] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 21:03:41 (1736993021) [ 2545.757176] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 21:03:45 (1736993025) [ 2564.050047] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 21:04:03 (1736993043) [ 2576.644568] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 21:04:15 (1736993055) [ 2585.471155] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 21:04:24 (1736993064) [ 2590.232122] bash (25788): drop_caches: 1 [ 2593.531849] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 21:04:32 (1736993072) [ 2595.334531] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 21:04:34 (1736993074) [ 2597.045862] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 21:04:36 (1736993076) [ 2598.691329] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 21:04:38 (1736993078) [ 2600.297873] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 21:04:39 (1736993079) [ 2600.502426] LustreError: 27493:0:(fld_handler.c:263:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 2602.292659] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 21:04:41 (1736993081) [ 2602.455750] LustreError: 27493:0:(fld_handler.c:263:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 2602.458492] LustreError: 27493:0:(fld_handler.c:263:fld_server_lookup()) Skipped 3 previous similar messages [ 2604.216292] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 21:04:43 (1736993083) [ 2605.864371] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 21:04:45 (1736993085) [ 2608.030758] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 21:04:47 (1736993087) [ 2609.898481] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 21:04:49 (1736993089) [ 2612.041580] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 21:04:51 (1736993091) [ 2730.030295] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 21:06:49 (1736993209) [ 2734.194715] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 21:06:53 (1736993213) [ 2738.212335] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 21:06:57 (1736993217) [ 2742.297067] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 21:07:01 (1736993221) [ 2746.436352] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 21:07:05 (1736993225) [ 2761.850389] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 21:07:21 (1736993241) [ 2782.966213] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 21:07:42 (1736993262) [ 2802.967614] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 21:08:02 (1736993282) [ 2821.177601] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 21:08:20 (1736993300) [ 2824.062623] Lustre: DEBUG MARKER: Turn on read and write cache [ 2825.406570] Lustre: DEBUG MARKER: Write data and read it back. [ 2825.835872] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 2827.102092] Lustre: DEBUG MARKER: cache hits: before: 65581, after: 65584 [ 2827.532476] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 2828.425129] Lustre: DEBUG MARKER: cache hits:: before: 65584, after: 65587 [ 2828.897888] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [ 2830.050234] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 2831.271089] Lustre: DEBUG MARKER: cache hits:: before: 65587, after: 65590 [ 2831.700517] Lustre: DEBUG MARKER: Write data and read it back. [ 2832.100858] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 2833.306071] Lustre: DEBUG MARKER: cache hits:: before: 65590, after: 65593 [ 2833.740754] Lustre: DEBUG MARKER: Turn off read and write cache [ 2834.883503] Lustre: DEBUG MARKER: Write data and read it back [ 2835.296011] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 2836.645031] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65593 [ 2837.082873] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [ 2838.380802] Lustre: DEBUG MARKER: Write data and read it back [ 2838.851248] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 2840.223670] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65593 [ 2840.707729] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 2842.070852] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65596 [ 2844.849442] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 21:08:44 (1736993324) [ 2845.777650] Lustre: lustre-MDD0000: changelog on [ 2846.700254] Lustre: lustre-MDD0001: changelog on [ 2851.678387] Lustre: Failing over lustre-MDT0000 [ 2851.748170] Lustre: server umount lustre-MDT0000 complete [ 2852.301661] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.538743] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2853.542482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.547956] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2854.255133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.303181] LustreError: 11-0: MGC192.168.203.137@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 2854.303193] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2854.303672] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf9415e9ba9 to 0x725a8bf941675057 [ 2854.315449] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 0@lo) [ 2854.378656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2854.388824] Lustre: lustre-MDD0000: changelog on [ 2854.393247] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2855.486901] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2857.303253] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2859.395845] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2859.446108] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2859.466073] Lustre: lustre-OST0001: deleting orphan objects from 0x0:17375 to 0x0:17409 [ 2859.466393] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17530 to 0x0:17569 [ 2859.733223] Lustre: lustre-MDD0000: changelog off [ 2860.345164] Lustre: lustre-MDD0001: changelog off [ 2864.791651] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 21:09:04 (1736993344) [ 2865.776767] Lustre: lustre-MDD0000: changelog on [ 2869.266111] Lustre: lustre-MDD0001: changelog off [ 2871.183115] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 21:09:10 (1736993350) [ 2872.143722] Lustre: lustre-MDD0000: changelog on [ 2872.145533] Lustre: Skipped 1 previous similar message [ 2876.196761] Lustre: lustre-MDD0001: changelog off [ 2876.198351] Lustre: Skipped 1 previous similar message [ 2878.016592] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 21:09:17 (1736993357) [ 2882.115616] Lustre: lustre-MDD0001: changelog off [ 2882.117240] Lustre: Skipped 1 previous similar message [ 2883.731805] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 21:09:23 (1736993363) [ 2884.618047] Lustre: lustre-MDD0000: changelog on [ 2884.619667] Lustre: Skipped 3 previous similar messages [ 2889.776521] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 21:09:29 (1736993369) [ 2893.771160] Lustre: DEBUG MARKER: 1736993373: creating first files [ 2910.068926] Lustre: *** cfs_fail_loc=1313, val=0*** [ 2910.070330] Lustre: 24964:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 2910.074421] Lustre: 18433:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl7 idle for 18s with 4 unprocessed records [ 2917.366811] Lustre: lustre-MDD0001: changelog off [ 2917.368378] Lustre: Skipped 3 previous similar messages [ 2919.166264] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 21:09:58 (1736993398) [ 2920.155128] Lustre: lustre-MDD0000: changelog on [ 2920.156788] Lustre: Skipped 3 previous similar messages [ 2927.076062] Lustre: 24964:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 2927.080077] Lustre: 24964:0:(mdd_dir.c:895:mdd_changelog_store()) Skipped 1 previous similar message [ 2927.084210] Lustre: 21719:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl9 idle for 5s with 4 unprocessed records [ 2927.089653] Lustre: 21719:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 2933.857102] Lustre: lustre-MDD0000: changelog off [ 2933.858841] Lustre: Skipped 2 previous similar messages [ 2934.587819] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 21:10:13 (1736993413) [ 2953.055765] Lustre: *** cfs_fail_loc=1316, val=0*** [ 2953.057555] Lustre: 24964:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0001: simulate starting changelog garbage collection [ 2953.061420] Lustre: 24964:0:(mdd_dir.c:895:mdd_changelog_store()) Skipped 1 previous similar message [ 2953.065404] Lustre: 25054:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0001: force deregister of changelog user cl11 idle for 15s with 3 unprocessed records [ 2953.071374] Lustre: 25054:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) Skipped 2 previous similar messages [ 2953.659886] Lustre: Failing over lustre-MDT0001 [ 2953.724530] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_disconnect to node 0@lo failed: rc = -107 [ 2954.546634] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.546795] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2954.555241] Lustre: Skipped 5 previous similar messages [ 2956.598852] Lustre: server umount lustre-MDT0001 complete [ 2959.058690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2959.109966] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2959.115036] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf941675057 to 0x725a8bf941676152 [ 2959.119995] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 0@lo) [ 2959.123191] Lustre: Skipped 3 previous similar messages [ 2959.199147] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2959.209924] LustreError: Skipped 4 previous similar messages [ 2959.218320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.232701] Lustre: lustre-MDD0000: changelog on [ 2959.234219] Lustre: Skipped 3 previous similar messages [ 2959.236144] Lustre: 26780:0:(mdd_device.c:618:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 27 to index 28, being cleared now [ 2959.243793] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2960.220472] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2962.476619] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2962.488331] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2962.532593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2962.644702] Lustre: 27759:0:(mdd_device.c:618:mdd_changelog_llog_init()) lustre-MDD0001 : orphan changelog records found, starting from index 19 to index 20, being cleared now [ 2962.652367] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 2963.605429] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2964.478897] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2964.674979] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2964.675017] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 2964.695381] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11209 to 0x280000400:11233 [ 2964.695663] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:11016 to 0x2c0000400:11041 [ 2964.704731] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17530 to 0x0:17601 [ 2964.708964] Lustre: lustre-OST0001: deleting orphan objects from 0x0:17411 to 0x0:17441 [ 2969.474658] Lustre: lustre-MDD0001: changelog off [ 2969.645651] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2969.648275] Lustre: Skipped 1 previous similar message [ 2971.038065] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 21:10:50 (1736993450) [ 2974.028244] LustreError: 30371:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 1315 sleeping [ 2976.230094] LustreError: 30540:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 1315 waking [ 2976.233063] LustreError: 30371:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 1315 awake: rc=2798 [ 2976.751618] LustreError: 30791:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 1315 waking [ 2982.942840] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 21:11:02 (1736993462) [ 2988.943158] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 21:11:08 (1736993468) [ 2991.056079] LustreError: 26792:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 15d sleeping for 3000ms [ 2994.059505] LustreError: 26792:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 15d awake [ 3001.384996] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 21:11:20 (1736993480) [ 3008.492882] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 21:11:27 (1736993487) [ 3012.283350] LustreError: 28376:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 15f sleeping [ 3014.285949] LustreError: 26791:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15f waking [ 3014.289017] LustreError: 26791:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message [ 3014.292021] LustreError: 28376:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 15f awake: rc=2994 [ 3018.966273] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 21:11:38 (1736993498) [ 3295.372865] LustreError: 31418:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 16c sleeping [ 3297.373901] LustreError: 28376:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 16c waking [ 3297.377378] LustreError: 31418:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 16c awake: rc=2997 [ 3299.358501] Lustre: lustre-MDD0001: changelog off [ 3299.360330] Lustre: Skipped 13 previous similar messages [ 3300.676547] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 21:16:20 (1736993780) [ 3301.397202] Lustre: lustre-MDD0000: changelog on [ 3301.398648] Lustre: Skipped 15 previous similar messages [ 3302.308650] LustreError: 8477:0:(mdd_device.c:1704:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [ 3302.526314] Lustre: 8541:0:(mdd_device.c:1721:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [ 3302.744829] LustreError: 8605:0:(mdd_device.c:1713:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [ 3309.788867] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 21:16:29 (1736993789) [ 3311.841890] Lustre: Failing over lustre-MDT0000 [ 3311.843524] Lustre: Skipped 1 previous similar message [ 3311.893502] Lustre: server umount lustre-MDT0000 complete [ 3311.894596] Lustre: Skipped 1 previous similar message [ 3311.929204] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.129912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3314.157391] LustreError: 11-0: MGC192.168.203.137@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 3314.157411] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.157413] Lustre: Skipped 1 previous similar message [ 3314.157509] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3314.157797] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf941676152 to 0x725a8bf941aad7be [ 3314.170766] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 0@lo) [ 3314.173769] Lustre: Skipped 3 previous similar messages [ 3314.218318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.220912] Lustre: Skipped 1 previous similar message [ 3314.263679] Lustre: 11580:0:(mdd_device.c:618:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 90151 to index 18446744073709551615, being cleared now [ 3314.271284] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3315.044785] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3316.932522] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3319.240291] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3319.242888] Lustre: Skipped 1 previous similar message [ 3319.256420] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17603 to 0x0:17633 [ 3319.256422] Lustre: lustre-OST0001: deleting orphan objects from 0x0:17443 to 0x0:17473 [ 3322.144073] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 21:16:41 (1736993801) [ 3325.030133] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records anaconda-ks.cfg stress.sh time ========================================================== 21:16:44 (1736993804) [ 3331.499498] Lustre: 26789:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 3331.502409] Lustre: 26789:0:(mdd_dir.c:895:mdd_changelog_store()) Skipped 1 previous similar message [ 3331.505724] Lustre: 14221:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl2 idle for 864005s with 500000004 unprocessed records [ 3336.436822] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 21:16:55 (1736993815) [ 3343.797716] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 21:17:03 (1736993823) [ 3350.958773] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 21:17:10 (1736993830) [ 3356.193519] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 21:17:15 (1736993835) [ 3362.893312] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 21:17:22 (1736993842) [ 3364.709840] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 21:17:24 (1736993844) [ 3366.491778] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 21:17:25 (1736993845) [ 3377.773421] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 21:17:37 (1736993857) [ 3383.713928] Lustre: Failing over lustre-OST0000 [ 3383.725631] Lustre: server umount lustre-OST0000 complete [ 3384.338789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.339052] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3384.339054] LustreError: Skipped 4 previous similar messages [ 3384.348202] Lustre: Skipped 4 previous similar messages [ 3388.207432] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3388.210975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3388.248084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.252557] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3389.157078] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3389.301801] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3389.564947] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3389.564956] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.137@tcp (at 0@lo) [ 3389.564957] Lustre: Skipped 4 previous similar messages [ 3389.566336] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11239 to 0x280000400:11265 [ 3389.566893] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17637 to 0x0:17665 [ 3390.096963] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 21:17:49 (1736993869) [ 3413.056614] Lustre: Failing over lustre-OST0000 [ 3413.068898] Lustre: server umount lustre-OST0000 complete [ 3413.282741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.286863] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3413.291615] LustreError: Skipped 3 previous similar messages [ 3415.320117] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3415.323295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3415.362219] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3416.217210] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3416.861422] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3417.009907] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3417.011232] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11239 to 0x280000400:11297 [ 3417.011781] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17667 to 0x0:17697 [ 3417.158806] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 21:18:16 (1736993896) [ 3426.568746] Lustre: Failing over lustre-OST0000 [ 3426.582256] Lustre: server umount lustre-OST0000 complete [ 3427.026636] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3427.026652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.026653] Lustre: Skipped 1 previous similar message [ 3427.035477] LustreError: Skipped 1 previous similar message [ 3428.909378] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3428.913003] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3429.850909] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3430.785094] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.137@tcp (at 0@lo) [ 3430.785105] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3430.790067] Lustre: Skipped 3 previous similar messages [ 3430.792025] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17765 to 0x0:17793 [ 3430.797045] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11359 to 0x280000400:11393 [ 3430.801070] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 21:18:30 (1736993910) [ 3450.981925] Lustre: Failing over lustre-OST0000 [ 3450.993448] Lustre: server umount lustre-OST0000 complete [ 3452.155264] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.159271] LustreError: Skipped 3 previous similar messages [ 3453.159423] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3453.162585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3453.196856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.200579] Lustre: Skipped 1 previous similar message [ 3453.202346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3453.205482] Lustre: Skipped 2 previous similar messages [ 3453.207789] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3453.209997] Lustre: Skipped 1 previous similar message [ 3454.076526] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3454.850334] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3454.853364] Lustre: Skipped 1 previous similar message [ 3454.984314] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 21:18:54 (1736993934) [ 3455.115719] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3455.117488] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11395 to 0x280000400:11425 [ 3455.118147] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17796 to 0x0:17825 [ 3467.627868] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3467.631763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3468.525867] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3469.313691] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17796 to 0x0:17857 [ 3469.314626] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11395 to 0x280000400:11457 [ 3469.455629] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 21:19:08 (1736993948) [ 3479.647127] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3479.650404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3480.547134] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3481.476268] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 21:19:20 (1736993960) [ 3481.486277] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11395 to 0x280000400:11489 [ 3481.489136] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17796 to 0x0:17889 [ 3481.859741] Lustre: DEBUG MARKER: creating a 10 Mb file [ 3482.289221] Lustre: DEBUG MARKER: starting reads [ 3482.690824] Lustre: DEBUG MARKER: truncating the file [ 3483.136973] Lustre: DEBUG MARKER: killing dd [ 3483.513976] Lustre: DEBUG MARKER: removing the temporary file [ 3485.180603] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log =============================================================================== 21:19:24 (1736993964) [ 3486.868008] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 21:19:26 (1736993966) [ 3491.511776] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 21:19:30 (1736993970) [ 3491.911750] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [ 3492.381873] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 21:19:31 (1736993971) [ 3493.135866] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_module obdecho/obdecho [ 3497.547413] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 21:19:36 (1736993976) [ 3498.215617] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_module obdecho/obdecho [ 3498.251898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3503.142969] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 21:19:42 (1736993982) [ 3518.006102] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ========================================================================== 21:19:57 (1736993997) [ 3527.015718] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 21:20:06 (1736994006) [ 3527.333286] Lustre: *** cfs_fail_loc=148, val=0*** [ 3529.329629] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 21:20:08 (1736994008) [ 3531.905760] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 21:20:11 (1736994011) [ 3534.007922] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 21:20:13 (1736994013) [ 3541.529831] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 21:20:20 (1736994020) [ 3546.480755] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 21:20:25 (1736994025) [ 3550.729164] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 21:20:30 (1736994030) [ 3552.348474] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 21:20:31 (1736994031) [ 3555.136928] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 21:20:34 (1736994034) [ 3560.989647] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 21:20:40 (1736994040) [ 3562.939776] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 21:20:42 (1736994042) [ 3564.590229] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 21:20:44 (1736994044) [ 3580.545286] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 21:20:59 (1736994059) [ 3582.292928] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 21:21:01 (1736994061) [ 3584.527201] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 21:21:03 (1736994063) [ 3586.277564] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 21:21:05 (1736994065) [ 3588.586930] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 21:21:07 (1736994067) [ 3590.542118] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 21:21:09 (1736994069) [ 3592.748464] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 21:21:11 (1736994071) [ 3594.974700] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 21:21:14 (1736994074) [ 3597.127878] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 21:21:16 (1736994076) [ 3600.768720] Lustre: lustre-MDD0000: changelog on [ 3600.770945] Lustre: Skipped 10 previous similar messages [ 3603.136379] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [ 3603.492734] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.15342 [ 3604.199807] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [ 3604.766281] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.3707 [ 3605.731887] Lustre: DEBUG MARKER: Test: lfs mkdir -i 1 /mnt/lustre/d205a.sanity.remote [ 3606.273341] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.21854 [ 3607.190663] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [ 3607.600615] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.9355 [ 3608.238918] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [ 3608.677879] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.23240 [ 3609.333096] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [ 3609.711726] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.13112 [ 3610.347729] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3610.731117] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.24291 [ 3611.580452] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [ 3612.120755] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.13792 [ 3612.925327] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [ 3613.319120] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.14893 [ 3614.027806] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [ 3614.420616] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.3329 [ 3615.428984] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [ 3615.979300] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.24748 [ 3616.713649] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [ 3617.134934] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.17141 [ 3621.058799] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3621.509581] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg337-client.v [ 3622.356240] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3622.786827] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg337-client.E [ 3623.436251] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3623.825331] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.oleg337-client.v [ 3625.978476] Lustre: lustre-MDD0001: changelog off [ 3625.980154] Lustre: Skipped 11 previous similar messages [ 3627.573674] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 21:21:46 (1736994106) [ 3630.491479] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 21:21:49 (1736994109) [ 3632.201699] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 21:21:51 (1736994111) [ 3633.868840] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 21:21:53 (1736994113) [ 3635.542314] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 21:21:54 (1736994114) [ 3637.179290] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 21:21:56 (1736994116) [ 3643.883250] Lustre: Failing over lustre-MDT0000 [ 3643.884466] Lustre: Skipped 2 previous similar messages [ 3643.950801] Lustre: server umount lustre-MDT0000 complete [ 3643.952236] Lustre: Skipped 2 previous similar messages [ 3644.770799] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3644.774198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3644.779019] Lustre: Skipped 5 previous similar messages [ 3644.780544] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3644.784695] LustreError: Skipped 4 previous similar messages [ 3651.954573] Lustre: 23107:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736994125/real 1736994125] req@ffff88012f458980 x1821368855927680/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1736994132 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 3651.963003] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3656.263501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3657.979296] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf941aad7be to 0x725a8bf941bf7268 [ 3657.982850] Lustre: MGC192.168.203.137@tcp: Connection restored to 192.168.203.137@tcp (at 0@lo) [ 3657.986127] Lustre: Skipped 7 previous similar messages [ 3658.035606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3658.037786] Lustre: Skipped 2 previous similar messages [ 3658.045541] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3658.047541] Lustre: Skipped 2 previous similar messages [ 3658.895616] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3659.492524] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3659.494760] Lustre: Skipped 2 previous similar messages [ 3663.049396] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3663.054168] Lustre: Skipped 2 previous similar messages [ 3663.068195] Lustre: lustre-OST0001: deleting orphan objects from 0x0:21031 to 0x0:21057 [ 3663.068210] Lustre: lustre-OST0000: deleting orphan objects from 0x0:21360 to 0x0:21377 [ 3664.406086] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3664.769724] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3679.770484] Lustre: 23105:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736994153/real 1736994153] req@ffff88012b92a140 x1821368855939520/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1736994160 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 3679.783403] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3681.216653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3685.803771] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf941bf7268 to 0x725a8bf941bf7689 [ 3687.050504] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3690.920583] Lustre: lustre-OST0000: deleting orphan objects from 0x0:21360 to 0x0:21409 [ 3690.920585] Lustre: lustre-OST0001: deleting orphan objects from 0x0:21031 to 0x0:21089 [ 3692.215975] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3692.590041] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3695.478670] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 21:22:54 (1736994174) [ 3703.990783] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 21:23:03 (1736994183) [ 3707.629461] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 21:23:07 (1736994187) [ 3709.540097] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 21:23:08 (1736994188) [ 3721.245446] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 21:23:20 (1736994200) [ 3725.355066] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 21:23:24 (1736994204) [ 3727.053547] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 21:23:26 (1736994206) [ 3732.542108] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') ========================================================== 21:23:31 (1736994211) [ 3734.617116] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 21:23:34 (1736994214) [ 3734.983452] Lustre: DEBUG MARKER: creating a 10 Mb file [ 3735.412555] Lustre: DEBUG MARKER: starting reads [ 3735.806351] Lustre: DEBUG MARKER: truncating the file [ 3736.216949] Lustre: DEBUG MARKER: killing dd [ 3736.595177] Lustre: DEBUG MARKER: removing the temporary file [ 3738.245708] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 21:23:37 (1736994217) [ 3739.978191] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 21:23:39 (1736994219) [ 3741.906679] Lustre: *** cfs_fail_loc=229, val=0*** [ 3743.874743] Lustre: *** cfs_fail_loc=229, val=1*** [ 3745.130004] Lustre: *** cfs_fail_loc=229, val=0*** [ 3751.680865] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 21:23:51 (1736994231) [ 3753.724674] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 21:23:53 (1736994233) [ 3755.628061] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 21:23:55 (1736994235) [ 3757.495274] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 21:23:56 (1736994236) [ 3759.425374] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 21:23:58 (1736994238) [ 3759.488575] Lustre: lustre-OST0001: Client 4c323211-16d2-4fff-b333-2ebeae4707d2 (at 192.168.203.37@tcp) reconnecting [ 3762.573676] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 21:24:01 (1736994241) [ 3764.043075] LustreError: 27246:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 sleeping for 3000ms [ 3767.049463] LustreError: 27246:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 awake [ 3770.115674] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 21:24:09 (1736994249) [ 3776.556438] Lustre: *** cfs_fail_loc=520, val=57344*** [ 3776.557219] LNet: *** cfs_fail_loc=e000, val=2147483648*** [ 3781.569192] Lustre: lustre-MDT0000: Client 4c323211-16d2-4fff-b333-2ebeae4707d2 (at 192.168.203.37@tcp) reconnecting [ 3781.581424] Lustre: 15255:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff880068642850 x1821368879135040/t34359738617(0) o35->4c323211-16d2-4fff-b333-2ebeae4707d2@192.168.203.37@tcp:0/0 lens 392/456 e 0 to 0 dl 1736994266 ref 1 fl Interpret:/2/0 rc 0/0 job:'' [ 3788.756795] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 21:24:28 (1736994268) [ 3789.826654] LustreError: 27246:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 sleeping for 22000ms [ 3809.621337] Lustre: lustre-OST0000: Client 4c323211-16d2-4fff-b333-2ebeae4707d2 (at 192.168.203.37@tcp) reconnecting [ 3810.931459] LustreError: 27246:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 3810.934252] LustreError: 27246:0:(ldlm_lib.c:3493:target_bulk_io()) @@@ bulk READ failed: rc = -107 req@ffff88008a538000 x1821368879140672/t0(0) o3->4c323211-16d2-4fff-b333-2ebeae4707d2@192.168.203.37@tcp:0/0 lens 488/440 e 0 to 0 dl 1736994290 ref 1 fl Interpret:/0/0 rc 0/0 job:'' [ 3810.942388] Lustre: 27246:0:(service.c:2333:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/1s); client may timeout req@ffff88008a538000 x1821368879140672/t0(0) o3->4c323211-16d2-4fff-b333-2ebeae4707d2@192.168.203.37@tcp:0/0 lens 488/440 e 0 to 0 dl 1736994290 ref 1 fl Complete:/0/0 rc 0/0 job:'' [ 3813.129785] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [ 3813.608090] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [ 3814.055230] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 21:24:53 (1736994293) [ 3815.821936] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 21:24:55 (1736994295) [ 3817.744366] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 21:24:57 (1736994297) [ 3817.850440] Lustre: lustre-MDT0000: subdir mount '/d226c.sanity' is remote and may be slow [ 3819.535626] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 21:24:58 (1736994298) [ 3821.173796] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 21:25:00 (1736994300) [ 3892.015556] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 21:26:11 (1736994371) [ 3969.459676] Lustre: Failing over lustre-MDT0000 [ 3969.462361] Lustre: Skipped 1 previous similar message [ 3969.517615] LustreError: 23105:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88012d01d580 x1821368857407808/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'osp-syn-0-0.0' [ 3969.525587] LustreError: 23105:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 3969.639765] Lustre: server umount lustre-MDT0000 complete [ 3969.642442] Lustre: Skipped 1 previous similar message [ 3970.355646] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3970.356090] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3970.356092] LustreError: Skipped 47 previous similar messages [ 3970.379931] Lustre: Skipped 10 previous similar messages [ 3972.982819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3973.054898] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3973.058369] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf941bf7689 to 0x725a8bf941d67b39 [ 3973.064390] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 0@lo) [ 3973.067897] Lustre: Skipped 9 previous similar messages [ 3973.172797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3973.176174] Lustre: Skipped 1 previous similar message [ 3973.201384] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3973.204914] Lustre: Skipped 1 previous similar message [ 3974.423428] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3974.880796] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3974.885901] Lustre: Skipped 1 previous similar message [ 3978.189144] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3978.194344] Lustre: Skipped 1 previous similar message [ 3978.218562] Lustre: lustre-OST0001: deleting orphan objects from 0x0:26097 to 0x0:26113 [ 3978.218631] Lustre: lustre-OST0000: deleting orphan objects from 0x0:26463 to 0x0:26497 [ 3998.580550] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 21:27:57 (1736994477) [ 4167.749331] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 21:30:47 (1736994647) [ 4169.917665] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 21:30:49 (1736994649) [ 4172.322324] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 21:30:51 (1736994651) [ 4186.088149] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 21:31:05 (1736994665) [ 4186.657698] Lustre: *** cfs_fail_loc=1801, val=0*** [ 4186.659505] Lustre: Skipped 1 previous similar message [ 4187.103901] LustreError: 5331:0:(mdd_dir.c:4234:mdd_migrate_cmd_check()) lustre-MDD0000: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush migrate_dir' to finish migration. [ 4189.636319] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [ 4190.210399] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 21:31:09 (1736994669) [ 4192.758373] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 21:31:12 (1736994672) [ 4195.428223] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 21:31:14 (1736994674) [ 4197.412777] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 21:31:16 (1736994676) [ 4199.516872] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 21:31:18 (1736994678) [ 4201.731913] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 21:31:21 (1736994681) [ 4203.915979] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 21:31:23 (1736994683) [ 4204.385465] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [ 4205.003151] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 21:31:24 (1736994684) [ 4233.118531] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 21:31:52 (1736994712) [ 4235.891175] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 21:31:55 (1736994715) [ 4238.016781] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 21:31:57 (1736994717) [ 4251.192829] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 21:32:10 (1736994730) [ 4263.889011] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 21:32:23 (1736994743) [ 4274.574755] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 21:32:33 (1736994753) [ 4276.572134] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 21:32:35 (1736994755) [ 4279.513026] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 21:32:38 (1736994758) [ 4281.396145] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 21:32:40 (1736994760) [ 4281.840403] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [ 4282.360296] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 21:32:41 (1736994761) [ 4282.786802] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [ 4283.284036] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 21:32:42 (1736994762) [ 4285.170602] Lustre: DEBUG MARKER: == sanity test 230y: unlink dir with bad hash type ======= 21:32:44 (1736994764) [ 4286.650093] Lustre: *** cfs_fail_loc=1802, val=0*** [ 4287.152767] Lustre: *** cfs_fail_loc=1802, val=0*** [ 4287.154445] Lustre: Skipped 35 previous similar messages [ 4290.928340] Lustre: DEBUG MARKER: == sanity test 230z: resume dir migration with bad hash type ========================================================== 21:32:50 (1736994770) [ 4292.442948] Lustre: *** cfs_fail_loc=1802, val=0*** [ 4292.444304] Lustre: Skipped 37 previous similar messages [ 4300.025422] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 21:32:59 (1736994779) [ 4303.109117] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 21:33:02 (1736994782) [ 4308.701013] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 21:33:08 (1736994788) [ 4308.979603] Lustre: *** cfs_fail_loc=31c, val=0*** [ 4310.199989] Lustre: Failing over lustre-OST0000 [ 4310.225388] Lustre: server umount lustre-OST0000 complete [ 4312.616956] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4312.621215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4312.663486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4312.671314] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4313.656154] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4314.029378] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4314.289696] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4314.289779] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.137@tcp (at 0@lo) [ 4314.289780] Lustre: Skipped 4 previous similar messages [ 4314.292624] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:32167 to 0x280000400:32193 [ 4314.293936] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28058 to 0x0:28097 [ 4316.073342] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 21:33:15 (1736994795) [ 4316.406662] Lustre: *** cfs_fail_loc=31c, val=0*** [ 4316.408550] LustreError: 27242:0:(ldlm_request.c:490:ldlm_cli_enqueue_local()) ### delayed lvb init failed (rc -12) ns: filter-lustre-OST0000_UUID lock: ffff88007d0cb180/0x725a8bf941f4ce40 lrc: 2/0,0 mode: --/PR res: [0x6dc2:0x0:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 27242 timeout: 0 lvb_type: 0 [ 4317.683057] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4317.687432] LustreError: Skipped 8 previous similar messages [ 4319.985050] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4319.989459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4321.140416] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4321.847845] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:32167 to 0x280000400:32225 [ 4321.851267] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28099 to 0x0:28129 [ 4323.668085] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 21:33:23 (1736994803) [ 4325.451914] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 21:33:24 (1736994804) [ 4327.343945] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 21:33:26 (1736994806) [ 4329.217430] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 21:33:28 (1736994808) [ 4333.072551] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 21:33:32 (1736994812) [ 4334.957028] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 21:33:34 (1736994814) [ 4336.819431] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 21:33:36 (1736994816) [ 4356.765015] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 21:33:56 (1736994836) [ 4357.724882] Lustre: *** cfs_fail_loc=2100, val=0*** [ 4363.965532] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 21:34:03 (1736994843) [ 4369.909065] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 21:34:09 (1736994849) [ 4370.600443] Lustre: *** cfs_fail_loc=713, val=0*** [ 4370.602273] Lustre: Skipped 1 previous similar message [ 4375.443224] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 21:34:14 (1736994854) [ 4396.366162] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 21:34:35 (1736994875) [ 4405.409257] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 21:34:44 (1736994884) [ 4405.777943] Lustre: *** cfs_fail_loc=105, val=0*** [ 4407.882454] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 21:34:47 (1736994887) [ 4424.237536] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 21:35:03 (1736994903) [ 4436.247898] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 21:35:15 (1736994915) [ 4438.257190] Lustre: DEBUG MARKER: == sanity test 245: check mdc connection flag/data: multiple modify RPCs ========================================================== 21:35:17 (1736994917) [ 4440.195556] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 21:35:19 (1736994919) [ 4442.720953] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 21:35:22 (1736994922) [ 4444.796016] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 21:35:24 (1736994924) [ 4445.079683] Lustre: lustre-MDT0000: subdir mount '/d247c.sanity' is remote and may be slow [ 4447.240375] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 21:35:26 (1736994926) [ 4447.546142] Lustre: lustre-MDT0000: subdir mount '/d247d.sanity' is remote and may be slow [ 4449.810037] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 21:35:29 (1736994929) [ 4452.063911] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 21:35:31 (1736994931) [ 4453.331014] Lustre: lustre-MDT0000: subdir mount '/d247f.sanity/remote' refused because 'enable_remote_subdir_mount=0': rc = -66 [ 4455.886104] Lustre: lustre-MDT0000: subdir mount '/d247f.sanity/remote' is remote and may be slow [ 4458.217414] Lustre: DEBUG MARKER: == sanity test 247g: mount striped directory as fileset caches ROOT lookup lock ========================================================== 21:35:37 (1736994937) [ 4458.645326] Lustre: DEBUG MARKER: SKIP: sanity test_247g needs >= 4 MDTs [ 4459.137272] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 21:35:38 (1736994938) [ 4466.495307] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 21:35:45 (1736994945) [ 4474.598438] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 21:35:54 (1736994954) [ 4476.364147] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 21:35:55 (1736994955) [ 4478.323318] Lustre: DEBUG MARKER: == sanity test 251: Handling short read and write correctly ========================================================== 21:35:57 (1736994957) [ 4480.445889] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 21:35:59 (1736994959) [ 4483.619384] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 21:36:02 (1736994962) [ 4534.287829] LustreError: 31418:0:(lod_qos.c:1362:lod_ost_alloc_specific()) can't lstripe objid [0x200004283:0x29:0x0]: have 0 want 1 [ 4580.867015] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 21:37:40 (1736995060) [ 4581.926716] Lustre: lustre-MDD0000: changelog on [ 4581.928336] Lustre: Skipped 1 previous similar message [ 4585.244249] Lustre: lustre-MDD0001: changelog off [ 4585.245504] Lustre: Skipped 1 previous similar message [ 4586.751703] Lustre: DEBUG MARKER: SKIP: sanity test_255a skipping excluded test 255a (base 255) [ 4587.276486] Lustre: DEBUG MARKER: SKIP: sanity test_255b skipping excluded test 255b (base 255) [ 4587.835159] Lustre: DEBUG MARKER: SKIP: sanity test_255c skipping excluded test 255c (base 255) [ 4588.370831] Lustre: DEBUG MARKER: SKIP: sanity test_256 skipping excluded test 256 [ 4588.855870] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 21:37:48 (1736995068) [ 4589.303934] Lustre: *** cfs_fail_loc=161, val=0*** [ 4589.829223] Lustre: Failing over lustre-MDT0001 [ 4589.830809] Lustre: Skipped 1 previous similar message [ 4590.498912] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4590.499037] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4590.499039] Lustre: Skipped 3 previous similar messages [ 4590.507057] Lustre: Skipped 5 previous similar messages [ 4591.109445] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.37@tcp (stopping) [ 4591.112823] Lustre: Skipped 2 previous similar messages [ 4595.506767] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4595.508783] Lustre: Skipped 2 previous similar messages [ 4595.521804] Lustre: server umount lustre-MDT0001 complete [ 4595.523018] Lustre: Skipped 1 previous similar message [ 4597.888250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4597.984115] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4597.993629] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 4597.998360] Lustre: Skipped 1 previous similar message [ 4598.920778] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4599.013629] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4599.016363] Lustre: Skipped 1 previous similar message [ 4600.136416] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 4602.999457] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4603.003220] Lustre: Skipped 1 previous similar message [ 4603.018941] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34730 to 0x280000400:34753 [ 4603.019218] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:34276 to 0x2c0000400:34305 [ 4606.872988] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 21:38:06 (1736995086) [ 4608.653052] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 21:38:08 (1736995088) [ 4610.376471] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 21:38:09 (1736995089) [ 4621.897216] Lustre: *** cfs_fail_loc=2301, val=0*** [ 4625.226466] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4625.232774] LDISKFS-fs (dm-2): 1 truncate cleaned up [ 4625.234376] LDISKFS-fs (dm-2): recovery complete [ 4625.236506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4625.279395] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4626.259819] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4626.646378] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34730 to 0x280000400:34785 [ 4626.646939] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28153 to 0x0:28193 [ 4630.948940] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 21:38:30 (1736995110) [ 4632.734879] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 21:38:32 (1736995112) [ 4636.256002] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 21:38:35 (1736995115) [ 4638.278883] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 21:38:37 (1736995117) [ 4640.320924] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 21:38:39 (1736995119) [ 4642.323532] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 21:38:41 (1736995121) [ 4644.434478] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 21:38:43 (1736995123) [ 4645.962479] Lustre: Increasing provided stripe size to a minimum value 64 [ 4649.220636] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 21:38:48 (1736995128) [ 4651.161077] Lustre: *** cfs_fail_loc=168, val=0*** [ 4651.687746] Lustre: *** cfs_fail_loc=168, val=0*** [ 4651.689571] Lustre: Skipped 2 previous similar messages [ 4652.826403] Lustre: *** cfs_fail_loc=168, val=0*** [ 4652.828051] Lustre: Skipped 5 previous similar messages [ 4656.740630] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 21:38:56 (1736995136) [ 4659.481054] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 21:38:58 (1736995138) [ 4661.294650] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 21:39:00 (1736995140) [ 4663.074881] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 21:39:02 (1736995142) [ 4664.846711] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 21:39:04 (1736995144) [ 4666.602266] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 21:39:06 (1736995146) [ 4687.178731] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 21:39:26 (1736995166) [ 4689.070466] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 21:39:28 (1736995168) [ 4690.866864] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 21:39:30 (1736995170) [ 4693.532603] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 21:39:32 (1736995172) [ 4695.339028] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 21:39:34 (1736995174) [ 4697.924707] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 21:39:37 (1736995177) [ 4700.280404] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 21:39:39 (1736995179) [ 4702.696459] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 21:39:42 (1736995182) [ 4704.630868] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 21:39:44 (1736995184) [ 4706.527678] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 21:39:45 (1736995185) [ 4708.302933] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 21:39:47 (1736995187) [ 4708.575817] LustreError: 26805:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 16b sleeping for 2000ms [ 4710.578669] LustreError: 26805:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 16b awake [ 4712.225301] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 21:39:51 (1736995191) [ 4715.484087] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 21:39:54 (1736995194) [ 4716.786576] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4716.789654] LustreError: Skipped 1 previous similar message [ 4718.196091] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4718.199223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4718.239979] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4719.102624] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4719.401458] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34805 to 0x280000400:34849 [ 4719.404791] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28206 to 0x0:28225 [ 4720.103859] LustreError: 20208:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4722.321644] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4722.325099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4722.362365] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4723.189654] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4724.179524] LustreError: 22723:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 4724.180364] LustreError: 22729:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4724.185197] Lustre: 21543:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4724.189433] Lustre: 21543:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 4726.359367] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4726.363304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4726.405688] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4727.299742] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4728.274835] LustreError: 25311:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 4728.275959] LustreError: 25317:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4728.275961] LustreError: 25317:0:(obd_class.h:1061:obd_statfs()) Skipped 4 previous similar messages [ 4728.282681] Lustre: 24122:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4728.285751] Lustre: 24122:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 4728.289698] Lustre: 24122:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 4730.452923] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4730.456652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4731.351593] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4731.499412] Lustre: 23103:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736995204/real 1736995204] req@ffff88011d275580 x1821368862370560/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1736995211 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 4732.320375] LustreError: 27893:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 4732.321205] LustreError: 27899:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4732.321207] LustreError: 27899:0:(obd_class.h:1061:obd_statfs()) Skipped 5 previous similar messages [ 4732.328831] Lustre: 26688:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4732.332481] Lustre: 26688:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 4732.335971] Lustre: 26688:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 4734.540185] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4734.544147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4734.598175] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4734.601892] Lustre: Skipped 1 previous similar message [ 4735.533660] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4736.547068] LustreError: 30417:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 4736.548779] LustreError: 30423:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4736.548781] LustreError: 30423:0:(obd_class.h:1061:obd_statfs()) Skipped 6 previous similar messages [ 4736.555445] Lustre: 29264:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4736.559000] Lustre: 29264:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 4736.562964] Lustre: 29264:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 4738.595423] Lustre: 23103:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736995211/real 1736995211] req@ffff8800a54f1c80 x1821368862376960/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1736995218 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 4738.604832] Lustre: 23103:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4738.832776] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4738.837578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4739.796918] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4740.804256] LustreError: 455:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 4740.807532] Lustre: 31762:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4740.811382] Lustre: 31762:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 4740.815205] Lustre: 31762:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 4743.027256] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4743.031478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4744.011600] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4745.012555] LustreError: 3023:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4745.015214] LustreError: 3023:0:(obd_class.h:1061:obd_statfs()) Skipped 11 previous similar messages [ 4747.078448] Lustre: 23103:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736995220/real 1736995220] req@ffff88013711c280 x1821368862384704/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1736995227 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 4747.088001] Lustre: 23103:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4747.216081] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4747.219897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4748.162036] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4749.136236] LustreError: 5562:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 4749.139333] LustreError: 5562:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) Skipped 1 previous similar message [ 4749.141533] Lustre: 4392:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4749.144628] Lustre: 4392:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 5 previous similar messages [ 4749.147678] Lustre: 4392:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 4749.150399] Lustre: 4392:0:(ofd_obd.c:554:ofd_postrecov()) Skipped 1 previous similar message [ 4751.300157] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4751.304585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4751.334282] Lustre: lustre-OST0000: Not available for connect from 192.168.203.37@tcp (not set up) [ 4751.352668] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4751.356262] Lustre: Skipped 3 previous similar messages [ 4752.240714] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4755.378533] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4755.383451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4756.283742] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4756.435410] Lustre: 23103:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736995229/real 1736995229] req@ffff880096803dc0 x1821368862391744/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1736995236 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 4756.443535] Lustre: 23103:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4759.383523] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4759.387048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4760.257493] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4761.254971] LustreError: 13175:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4761.257759] LustreError: 13175:0:(obd_class.h:1061:obd_statfs()) Skipped 9 previous similar messages [ 4763.426398] Lustre: 23103:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736995236/real 1736995236] req@ffff880122e6e3c0 x1821368862398144/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1736995243 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 4763.436594] Lustre: 23103:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4763.440933] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4763.443952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4764.329334] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4765.312406] LustreError: 15708:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 4765.314556] LustreError: 15708:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) Skipped 3 previous similar messages [ 4765.316449] Lustre: 14525:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4765.319099] Lustre: 14525:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 11 previous similar messages [ 4765.322582] Lustre: 14525:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 4765.326063] Lustre: 14525:0:(ofd_obd.c:554:ofd_postrecov()) Skipped 3 previous similar messages [ 4767.444648] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4767.448709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4768.341080] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4771.465842] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4771.469485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4772.388628] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4772.509418] Lustre: 23103:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736995245/real 1736995245] req@ffff880096802140 x1821368862405312/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1736995252 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 4772.517032] Lustre: 23103:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4775.459673] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4775.463049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4776.391686] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4779.652285] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4779.655636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4780.635559] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4781.576961] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34805 to 0x280000400:34881 [ 4781.577749] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28206 to 0x0:28257 [ 4783.899937] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4783.904200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4783.952735] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4783.956110] Lustre: Skipped 7 previous similar messages [ 4784.855828] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4788.241127] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4788.245014] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4789.294570] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4792.297421] Lustre: 23103:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736995265/real 1736995265] req@ffff880092b96d40 x1821368862422912/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1736995272 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 4792.306103] Lustre: 23103:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4792.637195] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4792.640182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4793.587036] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4794.574671] LustreError: 1052:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4794.577211] LustreError: 1052:0:(obd_class.h:1061:obd_statfs()) Skipped 17 previous similar messages [ 4796.806724] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4796.810316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4797.756886] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4800.292926] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 21:41:19 (1736995279) [ 4800.852213] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28206 to 0x0:28289 [ 4800.853231] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34805 to 0x280000400:34913 [ 4802.209496] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 21:41:21 (1736995281) [ 4802.960769] LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 [ 4803.011825] LustreError: 4314:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 60c sleeping [ 4805.326265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4805.356764] LustreError: 4890:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 60c waking [ 4805.356966] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4805.357339] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf941d67b39 to 0x725a8bf941ffa3f9 [ 4805.366285] LustreError: 4314:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 60c awake: rc=2648 [ 4807.418507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4807.420963] Lustre: Skipped 2 previous similar messages [ 4808.329824] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4810.740526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4810.811417] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 4810.815401] LustreError: Skipped 1 previous similar message [ 4811.696797] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4812.906156] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 4816.442745] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28206 to 0x0:28321 [ 4816.445109] Lustre: lustre-OST0001: deleting orphan objects from 0x0:27701 to 0x0:27745 [ 4816.446238] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:34314 to 0x2c0000400:34337 [ 4816.447181] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34805 to 0x280000400:34945 [ 4819.662102] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 21:41:39 (1736995299) [ 4820.212741] LustreError: 4898:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 15e sleeping [ 4821.707767] LustreError: 7241:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4821.709816] LustreError: 4898:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 15e awake: rc=3505 [ 4822.714082] LustreError: 7241:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4825.032960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4825.061935] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4825.061973] LustreError: 11-0: MGC192.168.203.137@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 4825.068750] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x725a8bf941ffa3f9 to 0x725a8bf941ffaadd [ 4825.075384] LustreError: 7843:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4826.124483] LustreError: 7844:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4827.128484] LustreError: 7844:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4829.134915] LustreError: 7844:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4829.137602] LustreError: 7844:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message [ 4833.146870] LustreError: 7844:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4833.149352] LustreError: 7844:0:(libcfs_fail.h:180:cfs_race()) Skipped 3 previous similar messages [ 4835.975785] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4837.155615] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 4837.157498] Lustre: Skipped 22 previous similar messages [ 4837.172350] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28206 to 0x0:28353 [ 4837.174488] Lustre: lustre-OST0001: deleting orphan objects from 0x0:27701 to 0x0:27777 [ 4841.540040] LustreError: 7844:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4841.542374] LustreError: 7844:0:(libcfs_fail.h:180:cfs_race()) Skipped 6 previous similar messages [ 4844.349419] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 21:42:03 (1736995323) [ 4846.500499] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 21:42:05 (1736995325) [ 4868.775261] Lustre: DEBUG MARKER: == sanity test 300c: chown [ 4887.364485] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 21:42:46 (1736995366) [ 4889.391704] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ========================================================== 21:42:48 (1736995368) [ 4891.157715] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ========================================================== 21:42:50 (1736995370) [ 4892.837254] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ========================================================== 21:42:52 (1736995372) [ 4895.426246] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory ========================================================== 21:42:54 (1736995374) [ 4898.236672] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory ========================================================== 21:42:57 (1736995377) [ 4901.027113] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ============ 21:43:00 (1736995380) [ 4902.906641] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ======== 21:43:02 (1736995382) [ 4905.022324] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ========================================================== 21:43:04 (1736995384) [ 4906.963021] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS ========================================================== 21:43:06 (1736995386) [ 4907.399017] Lustre: DEBUG MARKER: SKIP: sanity test_300m Only for single MDT [ 4907.908154] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA ========================================================== 21:43:07 (1736995387) [ 4912.123844] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [ 4912.576097] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================================== 21:43:11 (1736995391) [ 4912.899696] Lustre: *** cfs_fail_loc=1704, val=0*** [ 4912.901022] LustreError: 6984:0:(out_handler.c:910:out_tx_end()) lustre-MDT0001-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 4913.411731] LustreError: 7884:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 4913.415372] LustreError: 7884:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 4914.627467] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ========================================================== 21:43:14 (1736995394) [ 4916.357667] Lustre: DEBUG MARKER: == sanity test 300r: test -1 striped directory =========== 21:43:15 (1736995395) [ 4916.388768] LustreError: 7884:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [ 4916.392492] LustreError: 7884:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [ 4918.088804] Lustre: DEBUG MARKER: == sanity test 300s: test lfs mkdir -c without -i ======== 21:43:17 (1736995397) [ 4919.871811] Lustre: DEBUG MARKER: == sanity test 300t: test max_mdt_stripecount ============ 21:43:19 (1736995399) [ 4922.359780] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ============= 21:43:21 (1736995401) [ 4924.194171] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open ========================================================== 21:43:23 (1736995403) [ 4926.064152] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ========================================================== 21:43:25 (1736995405) [ 4926.483006] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [ 4926.972665] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ========================================================== 21:43:26 (1736995406) [ 4945.280627] Lustre: DEBUG MARKER: == sanity test 312: make sure ZFS adjusts its block size by write pattern ========================================================== 21:43:44 (1736995424) [ 4945.771518] Lustre: DEBUG MARKER: SKIP: sanity test_312 the test only applies to zfs [ 4946.267322] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ========================================================== 21:43:45 (1736995425) [ 4946.598980] Lustre: *** cfs_fail_loc=720, val=0*** [ 4946.600779] LustreError: 23415:0:(osd_handler.c:2093:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [ 4946.604136] LustreError: 23415:0:(ofd_objects.c:987:ofd_object_punch()) lustre-OST0000: failed to stop transaction: rc = -5 [ 4948.765831] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ========================================================== 21:43:48 (1736995428) [ 4949.612712] Lustre: *** cfs_fail_loc=720, val=0*** [ 4949.612715] LustreError: 25200:0:(osd_handler.c:2093:osd_trans_stop()) lustre-OST0001: failed in transaction hook: rc = -5 [ 4949.612724] LustreError: 25200:0:(ofd_objects.c:1054:ofd_destroy()) lustre-OST0001 failed to stop transaction: -5 [ 4949.613272] LustreError: 25200:0:(ofd_dev.c:1837:ofd_destroy_hdl()) lustre-OST0001: error destroying object [0x100010000:0x71b1:0x0]: -5 [ 4949.624471] Lustre: Skipped 1 previous similar message [ 4958.324873] Lustre: *** cfs_fail_loc=720, val=0*** [ 4958.326455] Lustre: Skipped 1 previous similar message [ 4958.327668] LustreError: 23415:0:(osd_handler.c:2093:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [ 4958.330593] LustreError: 23415:0:(osd_handler.c:2093:osd_trans_stop()) Skipped 2 previous similar messages [ 4961.349714] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ============= 21:44:00 (1736995440) [ 4965.275519] Lustre: DEBUG MARKER: == sanity test 316: lfs migrate of file with large_xattr enabled ========================================================== 21:44:04 (1736995444) [ 4967.429855] Lustre: DEBUG MARKER: == sanity test 317: Verify blocks get correctly update after truncate ========================================================== 21:44:06 (1736995446) [ 4969.504848] Lustre: DEBUG MARKER: == sanity test 318: Verify async readahead tunables ====== 21:44:08 (1736995448) [ 4971.324581] Lustre: DEBUG MARKER: == sanity test 319: lost lease lock on migrate error ===== 21:44:10 (1736995450) [ 4978.219533] Lustre: DEBUG MARKER: == sanity test 350: force NID mismatch path to be exercised ========================================================== 21:44:17 (1736995457) [ 4978.615115] LNet: *** cfs_fail_loc=e001, val=0*** [ 4978.617031] LNet: 24949:0:(lib-ptl.c:184:lnet_try_match_md()) message from 12345-0@lo matched 1821368863019072 with NID mismatch 0@lo accepted (same peer ffff880137301e00) [ 5011.136042] Lustre: DEBUG MARKER: == sanity test 398a: direct IO should cancel lock otherwise lockless ========================================================== 21:44:50 (1736995490) [ 5012.969894] Lustre: DEBUG MARKER: == sanity test 398b: DIO and buffer IO race ============== 21:44:52 (1736995492) [ 5017.946317] Lustre: DEBUG MARKER: == sanity test 398c: run fio to test AIO ================= 21:44:57 (1736995497) [ 5028.530107] Lustre: DEBUG MARKER: == sanity test 398d: run aiocp to verify block size > stripe size ========================================================== 21:45:07 (1736995507) [ 5032.196726] Lustre: DEBUG MARKER: == sanity test 398e: O_Direct open cleared by fcntl doesn't cause hang ========================================================== 21:45:11 (1736995511) [ 5033.871667] Lustre: DEBUG MARKER: == sanity test 398f: verify aio handles ll_direct_rw_pages errors correctly ========================================================== 21:45:13 (1736995513) [ 5036.119869] Lustre: DEBUG MARKER: == sanity test 398g: verify parallel dio async RPC submission ========================================================== 21:45:15 (1736995515) [ 5036.634010] LustreError: 23415:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5036.638397] LustreError: 23415:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 7 previous similar messages [ 5038.635398] LustreError: 19686:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 5038.638574] LustreError: 19686:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 5038.885072] LustreError: 19956:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5040.887480] LustreError: 19956:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 5040.889919] LustreError: 19956:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 4 previous similar messages [ 5040.903372] LustreError: 19956:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5042.907476] LustreError: 19956:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 5044.930696] LustreError: 19956:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5044.934259] LustreError: 19956:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 5046.936469] LustreError: 19956:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 5046.939718] LustreError: 19956:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 5053.272561] LustreError: 19956:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5053.275912] LustreError: 19956:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 5055.278459] LustreError: 19956:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 5055.280857] LustreError: 19956:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 5057.305415] Lustre: DEBUG MARKER: == sanity test 398h: verify correctness of read [ 5060.809723] Lustre: DEBUG MARKER: == sanity test 398i: verify parallel dio handles ll_direct_rw_pages errors correctly ========================================================== 21:45:40 (1736995540) [ 5063.468814] Lustre: DEBUG MARKER: == sanity test 398j: test parallel dio where stripe size > rpc_size ========================================================== 21:45:42 (1736995542) [ 5067.265338] Lustre: DEBUG MARKER: == sanity test 398k: test enospc on first stripe ========= 21:45:46 (1736995546) [ 5082.958583] Lustre: DEBUG MARKER: SKIP: sanity test_398k 7023424 > 600000 skipping out-of-space test on OST0 [ 5083.463615] Lustre: DEBUG MARKER: == sanity test 398l: test enospc on intermediate stripe/RPC ========================================================== 21:46:02 (1736995562) [ 5088.336833] Lustre: DEBUG MARKER: SKIP: sanity test_398l 7003512 > 600000 skipping out-of-space test on OST0 [ 5088.815011] Lustre: DEBUG MARKER: == sanity test 398m: test RPC failures with parallel dio ========================================================== 21:46:08 (1736995568) [ 5089.153696] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5089.155812] Lustre: Skipped 5 previous similar messages [ 5090.169385] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5090.171386] Lustre: Skipped 3 previous similar messages [ 5092.170342] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5092.171838] Lustre: Skipped 2 previous similar messages [ 5095.180432] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5095.181939] Lustre: Skipped 4 previous similar messages [ 5099.184164] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5099.185910] Lustre: Skipped 3 previous similar messages [ 5110.197474] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5110.198980] Lustre: Skipped 6 previous similar messages [ 5134.215404] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5134.217073] Lustre: Skipped 11 previous similar messages [ 5166.495420] Lustre: *** cfs_fail_loc=20f, val=0*** [ 5166.496913] Lustre: Skipped 34 previous similar messages [ 5237.281465] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5237.281466] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5237.281471] Lustre: Skipped 52 previous similar messages [ 5237.285515] Lustre: Skipped 1 previous similar message [ 5315.281183] Lustre: DEBUG MARKER: == sanity test 398n: test append with parallel DIO ======= 21:49:54 (1736995794) [ 5318.668902] Lustre: DEBUG MARKER: == sanity test 399a: fake write should not be slower than normal write ========================================================== 21:49:58 (1736995798) [ 5331.711687] Lustre: *** cfs_fail_loc=238, val=0*** [ 5345.314806] Lustre: DEBUG MARKER: == sanity test 399b: fake read should not be slower than normal read ========================================================== 21:50:24 (1736995824) [ 5352.055863] Lustre: DEBUG MARKER: SKIP: sanity test_400a skipping excluded test 400a [ 5352.494855] Lustre: DEBUG MARKER: == sanity test 400b: packaged headers can be compiled ==== 21:50:31 (1736995831) [ 5354.131697] Lustre: DEBUG MARKER: == sanity test 401a: Verify if 'lctl list_param -R' can list parameters recursively ========================================================== 21:50:33 (1736995833) [ 5355.899787] Lustre: DEBUG MARKER: == sanity test 401b: Verify 'lctl get_param' set_param' continue after error ========================================================== 21:50:35 (1736995835) [ 5357.569713] Lustre: DEBUG MARKER: == sanity test 401c: Verify 'lctl set_param' without value fails in either format. ========================================================== 21:50:36 (1736995836) [ 5359.192145] Lustre: DEBUG MARKER: == sanity test 401d: Verify 'lctl set_param' accepts values containing '=' ========================================================== 21:50:38 (1736995838) [ 5360.921319] Lustre: DEBUG MARKER: == sanity test 401e: verify 'lctl get_param' works with NID in parameter ========================================================== 21:50:40 (1736995840) [ 5362.632767] Lustre: DEBUG MARKER: == sanity test 402: Return ENOENT to lod_generate_and_set_lovea ========================================================== 21:50:42 (1736995842) [ 5362.960984] Lustre: *** cfs_fail_loc=15c, val=0*** [ 5362.962915] Lustre: Skipped 8 previous similar messages [ 5362.964939] LustreError: 26792:0:(lod_lov.c:816:lod_gen_component_ea()) lustre-MDT0000-mdtlov: Can not locate [0x100000000:0x7a1f:0x0]: rc = -2 [ 5364.665793] Lustre: DEBUG MARKER: == sanity test 403: i_nlink should not drop to zero due to aliasing ========================================================== 21:50:44 (1736995844) [ 5366.448138] Lustre: DEBUG MARKER: == sanity test 404: validate manual {de}activated works properly for OSPs ========================================================== 21:50:45 (1736995845) [ 5366.989033] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5367.457329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.463070] Lustre: Skipped 16 previous similar messages [ 5367.465027] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5367.468229] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.474219] Lustre: lustre-OST0000: deleting orphan objects from 0x0:31264 to 0x0:31297 [ 5367.957078] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5368.430986] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5368.433229] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5368.439857] Lustre: lustre-OST0001: deleting orphan objects from 0x0:29675 to 0x0:29697 [ 5369.403850] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:36347 to 0x280000400:36385 [ 5369.877791] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5369.880149] Lustre: Skipped 1 previous similar message [ 5370.365857] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5370.368818] Lustre: Skipped 1 previous similar message [ 5370.370570] LustreError: 167-0: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5370.374901] LustreError: Skipped 1 previous similar message [ 5370.377728] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:35739 to 0x2c0000400:35777 [ 5372.313179] Lustre: DEBUG MARKER: == sanity test 405: Various layout swap lock tests ======= 21:50:51 (1736995851) [ 5375.716586] Lustre: DEBUG MARKER: SKIP: sanity test_405 layout swap does not support DOM files so far [ 5376.210970] Lustre: DEBUG MARKER: == sanity test 406: DNE support fs default striping ====== 21:50:55 (1736995855) [ 5391.663646] Lustre: DEBUG MARKER: SKIP: sanity test_407 skipping ALWAYS excluded test 407 [ 5392.178244] Lustre: DEBUG MARKER: == sanity test 408: drop_caches should not hang due to page leaks ========================================================== 21:51:11 (1736995871) [ 5396.080412] Lustre: DEBUG MARKER: == sanity test 409: Large amount of cross-MDTs hard links on the same file ========================================================== 21:51:15 (1736995875) [ 5414.266814] Lustre: DEBUG MARKER: == sanity test 410: Test inode number returned from kernel thread ========================================================== 21:51:33 (1736995893) [ 5416.139449] Lustre: DEBUG MARKER: == sanity test 411: Slab allocation error with cgroup does not LBUG ========================================================== 21:51:35 (1736995895) [ 5418.567515] Lustre: DEBUG MARKER: == sanity test 412: mkdir on specific MDTs =============== 21:51:37 (1736995897) [ 5420.745404] Lustre: DEBUG MARKER: == sanity test 413a: QoS mkdir with 'lfs mkdir -i -1' ==== 21:51:40 (1736995900) [ 5589.352763] Lustre: DEBUG MARKER: == sanity test 413b: QoS mkdir under dir whose default LMV starting MDT offset is -1 ========================================================== 21:54:28 (1736996068) [ 5629.924530] Lustre: DEBUG MARKER: == sanity test 413c: mkdir with default LMV max inherit rr ========================================================== 21:55:08 (1736996108) [ 5674.764699] Lustre: DEBUG MARKER: == sanity test 413d: inherit ROOT default LMV ============ 21:55:53 (1736996153) [ 5684.535026] Lustre: DEBUG MARKER: == sanity test 413e: check default max-inherit value ===== 21:56:03 (1736996163) [ 5688.423129] Lustre: DEBUG MARKER: == sanity test 413f: lfs getdirstripe -D list ROOT default LMV if it's not set on dir ========================================================== 21:56:07 (1736996167) [ 5692.415878] Lustre: DEBUG MARKER: == sanity test 413z: 413 test cleanup ==================== 21:56:11 (1736996171) [ 5706.567226] Lustre: DEBUG MARKER: == sanity test 414: simulate ENOMEM in ptlrpc_register_bulk() ========================================================== 21:56:25 (1736996185) [ 5710.273696] Lustre: DEBUG MARKER: == sanity test 415: lock revoke is not missing =========== 21:56:29 (1736996189) [ 5733.299279] Lustre: DEBUG MARKER: == sanity test 416: transaction start failure won't cause system hung ========================================================== 21:56:52 (1736996212) [ 5733.818580] Lustre: *** cfs_fail_loc=19a, val=0*** [ 5737.276672] Lustre: DEBUG MARKER: == sanity test 417: disable remote dir, striped dir and dir migration ========================================================== 21:56:56 (1736996216) [ 5747.632523] Lustre: DEBUG MARKER: == sanity test 418: df and lfs df outputs match ========== 21:57:06 (1736996226) [ 5784.395659] Lustre: DEBUG MARKER: sanity test_418: @@@@@@ FAIL: df and lfs df inodes output mismatch: df -i: 192.168.203.137@tcp:/lustre 532178 18548 513630 4% /mnt/lustre, lfs df -i: filesystem_summary: 527147 13517 513630 3% /mnt/lustre