[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] mapped at [ffffffffff2003f0] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5200 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1d87 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1c23 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1c97 00090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1d27 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1d5f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 350245905 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.551961] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.554313] pid_max: default: 32768 minimum: 301 [ 0.556041] Security Framework initialized [ 0.557065] SELinux: Initializing. [ 0.557973] SELinux: Starting in permissive mode [ 0.559365] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.563178] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.567064] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.570636] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.574527] Initializing cgroup subsys memory [ 0.575513] Initializing cgroup subsys devices [ 0.576289] Initializing cgroup subsys freezer [ 0.577118] Initializing cgroup subsys net_cls [ 0.578607] Initializing cgroup subsys blkio [ 0.582447] Initializing cgroup subsys perf_event [ 0.584331] Initializing cgroup subsys hugetlb [ 0.586223] Initializing cgroup subsys pids [ 0.588846] Initializing cgroup subsys net_prio [ 0.594602] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.597936] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.599917] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.602237] tlb_flushall_shift: 6 [ 0.603554] FEATURE SPEC_CTRL Present [ 0.605159] FEATURE IBPB_SUPPORT Present [ 0.606588] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.608983] Spectre V2 : Vulnerable [ 0.610309] Speculative Store Bypass: Vulnerable [ 0.613164] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.621239] ACPI: Core revision 20130517 [ 0.626016] ACPI: All ACPI Tables successfully acquired [ 0.628991] ftrace: allocating 30294 entries in 119 pages [ 0.696720] Enabling x2apic [ 0.697565] Enabled x2apic [ 0.701112] Switched APIC routing to physical x2apic. [ 0.706752] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.709005] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.713603] TSC deadline timer enabled [ 0.713842] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.715777] ... version: 2 [ 0.716828] ... bit width: 48 [ 0.718124] ... generic registers: 4 [ 0.719295] ... value mask: 0000ffffffffffff [ 0.720999] ... max period: 00007fffffffffff [ 0.722548] ... fixed-purpose events: 3 [ 0.723620] ... event mask: 000000070000000f [ 0.725314] KVM setup paravirtual spinlock [ 0.744433] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.749718] KVM setup async PF for cpu 1 [ 0.758766] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.775748] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.730053] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.782109] KVM setup async PF for cpu 2 [ 0.783231] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.792979] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.798203] Brought up 4 CPUs [ 0.798207] KVM setup async PF for cpu 3 [ 0.798215] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.802136] smpboot: Max logical packages: 1 [ 0.803477] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.813753] devtmpfs: initialized [ 0.837835] x86/mm: Memory block size: 128MB [ 0.849765] EVM: security.selinux [ 0.850828] EVM: security.ima [ 0.851572] EVM: security.capability [ 0.856114] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.857742] NET: Registered protocol family 16 [ 0.859376] cpuidle: using governor haltpoll [ 0.861596] ACPI: bus type PCI registered [ 0.864549] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.868441] PCI: Using configuration type 1 for base access [ 0.871100] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.891913] ACPI: Added _OSI(Module Device) [ 0.894796] ACPI: Added _OSI(Processor Device) [ 0.897402] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.900639] ACPI: Added _OSI(Processor Aggregator Device) [ 0.903864] ACPI: Added _OSI(Linux-Dell-Video) [ 0.913487] ACPI: EC: Look up EC in DSDT [ 0.920935] ACPI: Interpreter enabled [ 0.923193] ACPI: (supports S0 S3 S4 S5) [ 0.925487] ACPI: Using IOAPIC for interrupt routing [ 0.928465] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.933634] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.945046] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.948073] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.953305] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.957133] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.966757] acpiphp: Slot [2] registered [ 0.968307] acpiphp: Slot [5] registered [ 0.970425] acpiphp: Slot [6] registered [ 0.971920] acpiphp: Slot [7] registered [ 0.973601] acpiphp: Slot [8] registered [ 0.974990] acpiphp: Slot [9] registered [ 0.977307] acpiphp: Slot [10] registered [ 0.978993] acpiphp: Slot [3] registered [ 0.985884] acpiphp: Slot [4] registered [ 0.987263] acpiphp: Slot [11] registered [ 0.989420] acpiphp: Slot [12] registered [ 0.991141] acpiphp: Slot [13] registered [ 0.993263] acpiphp: Slot [14] registered [ 0.994566] acpiphp: Slot [15] registered [ 0.996154] acpiphp: Slot [16] registered [ 0.997578] acpiphp: Slot [17] registered [ 0.999042] acpiphp: Slot [18] registered [ 1.000480] acpiphp: Slot [19] registered [ 1.001682] acpiphp: Slot [20] registered [ 1.003062] acpiphp: Slot [21] registered [ 1.004316] acpiphp: Slot [22] registered [ 1.005583] acpiphp: Slot [23] registered [ 1.006850] acpiphp: Slot [24] registered [ 1.007805] acpiphp: Slot [25] registered [ 1.010367] acpiphp: Slot [26] registered [ 1.011845] acpiphp: Slot [27] registered [ 1.013255] acpiphp: Slot [28] registered [ 1.014673] acpiphp: Slot [29] registered [ 1.016088] acpiphp: Slot [30] registered [ 1.017656] acpiphp: Slot [31] registered [ 1.019368] PCI host bridge to bus 0000:00 [ 1.020525] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 1.023043] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 1.024933] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 1.027012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 1.029320] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 1.032331] pci_bus 0000:00: root bus resource [bus 00-ff] [ 1.034573] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 1.035370] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 1.036324] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 1.043014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 1.046298] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 1.047781] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 1.049770] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 1.051592] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 1.054321] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 1.054891] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 1.056083] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 1.057943] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 1.059219] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 1.069369] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 1.072010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 1.073551] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 1.081786] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 1.095316] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 1.112549] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 1.121238] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.124286] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.127242] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.142402] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 1.154370] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.159925] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.173078] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.198865] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 1.217669] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.238235] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.246090] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.268482] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 1.279068] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.282730] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.286219] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.300619] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 1.311255] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.315801] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.320108] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.332399] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 1.338761] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.341573] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.344939] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.347029] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.349083] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.352105] vgaarb: loaded [ 1.353753] SCSI subsystem initialized [ 1.355018] ACPI: bus type USB registered [ 1.356439] usbcore: registered new interface driver usbfs [ 1.358308] usbcore: registered new interface driver hub [ 1.361864] usbcore: registered new device driver usb [ 1.364136] PCI: Using ACPI for IRQ routing [ 1.365137] PCI: pci_cache_line_size set to 64 bytes [ 1.365308] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.365312] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.365314] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.365723] NetLabel: Initializing [ 1.366881] NetLabel: domain hash size = 128 [ 1.368168] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.370085] NetLabel: unlabeled traffic allowed by default [ 1.372346] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.374050] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.381049] amd_nb: Cannot enumerate AMD northbridges [ 1.386453] Switched to clocksource kvm-clock [ 1.424494] pnp: PnP ACPI init [ 1.425381] ACPI: bus type PNP registered [ 1.426630] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.426726] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.426772] pnp 00:02: [dma 2] [ 1.426816] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.426887] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.426961] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.427041] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.427521] pnp: PnP ACPI: found 6 devices [ 1.428624] ACPI: bus type PNP unregistered [ 1.444008] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.444015] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.444018] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.444021] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.444024] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.444125] NET: Registered protocol family 2 [ 1.446248] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.457079] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.464933] TCP: Hash tables configured (established 32768 bind 32768) [ 1.471814] TCP: reno registered [ 1.474373] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.478030] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.482213] NET: Registered protocol family 1 [ 1.486458] RPC: Registered named UNIX socket transport module. [ 1.490788] RPC: Registered udp transport module. [ 1.492099] RPC: Registered tcp transport module. [ 1.494884] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.497125] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.498623] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.500176] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.501914] PCI: CLS 0 bytes, default 64 [ 1.502102] Unpacking initramfs... [ 3.800712] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 3.805618] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.807992] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 3.811195] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 3.813610] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 3.815329] RAPL PMU: hw unit of domain package 2^-0 Joules [ 3.816641] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 3.855031] cryptomgr_test (52) used greatest stack depth: 14480 bytes left [ 3.857624] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 3.859305] Initialise system trusted keyring [ 3.933402] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 3.935561] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 3.945240] zpool: loaded [ 3.946535] zbud: loaded [ 3.949292] VFS: Disk quotas dquot_6.6.0 [ 3.953532] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 3.960641] NFS: Registering the id_resolver key type [ 3.964058] Key type id_resolver registered [ 3.968718] Key type id_legacy registered [ 3.972543] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 3.984166] Key type big_key registered [ 3.988684] SELinux: Registering netfilter hooks [ 3.995780] cryptomgr_test (58) used greatest stack depth: 14048 bytes left [ 4.001194] cryptomgr_test (59) used greatest stack depth: 13664 bytes left [ 4.007347] NET: Registered protocol family 38 [ 4.010785] Key type asymmetric registered [ 4.015853] Asymmetric key parser 'x509' registered [ 4.017297] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 4.026753] io scheduler noop registered [ 4.031160] io scheduler deadline registered (default) [ 4.036815] io scheduler cfq registered [ 4.039635] io scheduler mq-deadline registered [ 4.040902] io scheduler kyber registered [ 4.048934] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.050310] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.058031] intel_idle: does not run on family 6 model 62 [ 4.060812] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.066430] ACPI: Power Button [PWRF] [ 4.072143] GHES: HEST is not enabled! [ 4.149259] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 4.222044] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 4.380126] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 4.470913] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 4.686562] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.720986] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 4.748852] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 4.759245] Non-volatile memory driver v1.3 [ 4.761255] Linux agpgart interface v0.103 [ 4.763871] crash memory driver: version 1.1 [ 4.767201] nbd: registered device at major 43 [ 4.797294] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 4.799407] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 4.799482] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 4.799528] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 4.799571] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 4.810107] virtio_blk virtio1: [vda] 53432 512-byte logical blocks (27.3 MB/26.0 MiB) [ 4.828125] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 4.828166] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 4.828197] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 4.828232] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 4.828262] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 4.835811] tsc: Refined TSC clocksource calibration: 2399.956 MHz [ 4.838117] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.847892] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 4.847945] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 4.847988] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 4.848031] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 4.848071] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 4.856141] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.862946] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 4.862993] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 4.863039] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 4.863073] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 4.863111] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 4.871260] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.878265] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 4.878308] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 4.878868] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 4.878912] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 4.878957] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 4.889167] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.897321] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 4.897380] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 4.897418] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 4.897457] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 4.897502] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 4.910681] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.922760] rdac: device handler registered [ 4.924431] hp_sw: device handler registered [ 4.925764] emc: device handler registered [ 4.927423] libphy: Fixed MDIO Bus: probed [ 4.946521] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 4.949576] ehci-pci: EHCI PCI platform driver [ 4.951135] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 4.953285] ohci-pci: OHCI PCI platform driver [ 4.955627] uhci_hcd: USB Universal Host Controller Interface driver [ 4.957934] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.962792] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.964048] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.966778] mousedev: PS/2 mouse device common for all mice [ 4.973307] rtc_cmos 00:05: RTC can wake from S4 [ 4.975153] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 4.976199] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.979703] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.991035] hidraw: raw HID events driver (C) Jiri Kosina [ 4.993091] usbcore: registered new interface driver usbhid [ 4.997576] usbhid: USB HID core driver [ 4.999408] drop_monitor: Initializing network drop monitor service [ 5.002161] Netfilter messages via NETLINK v0.30. [ 5.006314] TCP: cubic registered [ 5.007244] Initializing XFRM netlink socket [ 5.008956] NET: Registered protocol family 10 [ 5.011067] NET: Registered protocol family 17 [ 5.012443] Key type dns_resolver registered [ 5.014310] mce: Using 10 MCE banks [ 5.015722] Loading compiled-in X.509 certificates [ 5.018341] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 5.021904] registered taskstats version 1 [ 5.031924] modprobe (72) used greatest stack depth: 13456 bytes left [ 5.045838] Key type trusted registered [ 5.060254] Key type encrypted registered [ 5.061746] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 5.065950] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 5.074469] rtc_cmos 00:05: setting system clock to 2025-05-14 21:06:27 UTC (1747256787) [ 5.077858] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 5.083315] Write protecting the kernel read-only data: 12288k [ 5.087086] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 5.089868] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 5.104451] random: systemd: uninitialized urandom read (16 bytes read) [ 5.109068] random: systemd: uninitialized urandom read (16 bytes read) [ 5.111045] random: systemd: uninitialized urandom read (16 bytes read) [ 5.115987] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 5.122669] systemd[1]: Detected virtualization kvm. [ 5.124565] systemd[1]: Detected architecture x86-64. [ 5.126560] systemd[1]: Running in initial RAM disk. [ 5.131637] systemd[1]: No hostname configured. [ 5.132648] systemd[1]: Set hostname to . [ 5.134193] random: systemd: uninitialized urandom read (16 bytes read) [ 5.135654] systemd[1]: Initializing machine ID from random generator. [ 5.223092] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 5.229459] random: systemd: uninitialized urandom read (16 bytes read) [ 5.231499] random: systemd: uninitialized urandom read (16 bytes read) [ 5.233415] random: systemd: uninitialized urandom read (16 bytes read) [ 5.235353] random: systemd: uninitialized urandom read (16 bytes read) [ 5.249985] random: systemd: uninitialized urandom read (16 bytes read) [ 5.258726] random: systemd: uninitialized urandom read (16 bytes read) [ 5.278072] systemd[1]: Reached target Swap. [ 5.282743] systemd[1]: Created slice Root Slice. [ 5.285685] systemd[1]: Listening on udev Kernel Socket. [ 5.291897] systemd[1]: Created slice System Slice. [ 5.296404] systemd[1]: Listening on Journal Socket. [ 5.307151] systemd[1]: Starting Setup Virtual Console... [ 5.315464] systemd[1]: Starting Journal Service... [ 5.323444] systemd[1]: Starting dracut cmdline hook... [ 5.337333] systemd[1]: Starting Load Kernel Modules... [ 5.342768] systemd[1]: Listening on udev Control Socket. [ 5.350607] systemd[1]: Reached target Sockets. [ 5.367441] systemd[1]: Reached target Slices. [ 5.372857] systemd[1]: Reached target Local File Systems. [ 5.378281] systemd[1]: Reached target Timers. [ 5.384685] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 5.394727] systemd[1]: Started Journal Service. [ 5.915808] random: fast init done [ 5.921354] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 6.306123] libata version 3.00 loaded. [ 6.317681] ata_piix 0000:00:01.1: version 2.13 [ 6.325032] scsi host0: ata_piix [ 6.326600] scsi host1: ata_piix [ 6.330303] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.337656] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 6.555294] ip (322) used greatest stack depth: 13080 bytes left [ 6.617674] ip (345) used greatest stack depth: 12344 bytes left [ 9.814599] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 10.410004] systemd-journald[102]: Received SIGTERM from PID 1 (systemd). [ 10.743683] SELinux: Disabled at runtime. [ 10.745540] SELinux: Unregistering netfilter hooks [ 10.839159] ip_tables: (C) 2000-2006 Netfilter Core Team [ 10.843703] systemd[1]: Inserted module 'ip_tables' [ 12.076444] systemd-journald[571]: Received request to flush runtime journal from PID 1 [ 12.392676] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.442429] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 12.519761] cryptd: max_cpu_qlen set to 1000 [ 12.713682] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 12.762308] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 12.800924] AVX version of gcm_enc/dec engaged. [ 12.802618] AES CTR mode by8 optimization enabled [ 12.876513] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 12.882924] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 13.243208] EDAC MC: Ver: 3.0.0 [ 13.306713] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 13.306725] EDAC sbridge: Ver: 1.1.2 [ 18.162696] mount.nfs (778) used greatest stack depth: 10704 bytes left [ 29.678404] device-mapper: uevent: version 1.0.3 [ 29.680535] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 34.971826] libcfs: loading out-of-tree module taints kernel. [ 34.973982] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 34.982068] alg: No test for adler32 (adler32-zlib) [ 35.758961] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 41.098463] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 41.337744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 41.521277] Lustre: Lustre: Build Version: 2.15.6_28_g2a10569 [ 41.720416] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 41.722632] LNet: Accept secure, port 988 [ 43.287412] Key type lgssc registered [ 43.617026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 46.855195] icp: module license 'CDDL' taints kernel. [ 46.857137] Disabling lock debugging due to kernel taint [ 48.386801] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 52.301378] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 59.554089] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 63.847174] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 63.856887] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 67.267719] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 67.273480] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 71.725112] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 76.327233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 76.360568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 77.457812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 77.472459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 77.506161] Lustre: lustre-MDT0000: new disk, initializing [ 77.529681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 77.538999] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 77.569744] mount.lustre (6830) used greatest stack depth: 10336 bytes left [ 78.718309] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 83.085763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.105152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 83.123685] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 83.129922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 83.131604] Lustre: Skipped 1 previous similar message [ 83.161975] Lustre: lustre-MDT0001: new disk, initializing [ 83.175916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 83.184365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 83.187578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 84.092581] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 88.373928] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 88.376881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 88.396605] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 88.400170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 88.402593] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 88.460471] Lustre: lustre-OST0000: new disk, initializing [ 88.461708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 88.471964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 89.340522] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 90.383008] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 90.386791] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 94.233440] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 94.239484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.268095] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 94.275260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 94.281392] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 94.317094] Lustre: lustre-OST0001: new disk, initializing [ 94.319138] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 94.337902] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 95.585203] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 98.445005] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 98.448776] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 101.383180] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 107.118407] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 112.861841] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing check_logdir /tmp/testlogs/ [ 113.312721] random: crng init done [ 113.902909] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing yml_node [ 115.432571] Lustre: DEBUG MARKER: Client: 2.15.6.28 [ 116.405296] Lustre: DEBUG MARKER: MDS: 2.15.6.28 [ 117.344279] Lustre: DEBUG MARKER: OSS: 2.15.6.28 [ 117.904998] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Wed May 14 17:08:20 EDT 2025 [ 120.867207] Lustre: DEBUG MARKER: - need mds1 < v2_14_55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 121.377218] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42b 42c 407 118c 118d [ 121.907272] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 115 135 136 230d 300o [ 123.764024] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing check_config_client /mnt/lustre [ 130.477884] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 131.727997] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 132.925002] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 136.505738] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 17:08:39 (1747256919) [ 137.819255] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 138.629820] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 17:08:41 (1747256921) [ 141.444889] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 17:08:43 (1747256923) [ 188.052307] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 17:09:30 (1747256970) [ 190.255410] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 17:09:32 (1747256972) [ 190.655578] Lustre: *** cfs_fail_loc=15b, val=0*** [ 190.657592] Lustre: *** cfs_fail_loc=15b, val=0*** [ 193.022631] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 17:09:35 (1747256975) [ 195.880204] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 17:09:38 (1747256978) [ 196.393450] Lustre: *** cfs_fail_loc=19a, val=0*** [ 197.082649] Lustre: *** cfs_fail_loc=19a, val=0*** [ 197.085091] Lustre: Skipped 1 previous similar message [ 197.088378] LustreError: 6881:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 197.093157] LustreError: 6881:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 198.108150] Lustre: *** cfs_fail_loc=19a, val=0*** [ 198.111023] Lustre: Skipped 2 previous similar messages [ 198.113074] LustreError: 8084:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0001-osd: fail to cancel 1 llog-records: rc = -5 [ 198.119063] LustreError: 8084:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message [ 198.125864] LustreError: 8084:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0001-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 198.130584] LustreError: 8084:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1 previous similar message [ 199.160237] LustreError: 6881:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 199.165240] LustreError: 6881:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message [ 199.169644] LustreError: 6881:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 199.174571] LustreError: 6881:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1 previous similar message [ 200.443086] Lustre: *** cfs_fail_loc=19a, val=0*** [ 200.445192] Lustre: Skipped 6 previous similar messages [ 201.431573] LustreError: 8084:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0001-osd: fail to cancel 1 llog-records: rc = -5 [ 201.438819] LustreError: 8084:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 4 previous similar messages [ 201.442998] LustreError: 8084:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0001-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 201.450767] LustreError: 8084:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 4 previous similar messages [ 204.570655] Lustre: *** cfs_fail_loc=19a, val=0*** [ 204.572512] Lustre: Skipped 11 previous similar messages [ 205.569432] LustreError: 8084:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0001-osd: fail to cancel 1 llog-records: rc = -5 [ 205.574632] LustreError: 8084:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 7 previous similar messages [ 205.577797] LustreError: 8084:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0001-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 205.581928] LustreError: 8084:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 7 previous similar messages [ 212.728575] Lustre: *** cfs_fail_loc=19a, val=0*** [ 212.731383] Lustre: Skipped 26 previous similar messages [ 214.077717] LustreError: 6881:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 214.081873] LustreError: 6881:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 12 previous similar messages [ 214.085173] LustreError: 6881:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 214.091281] LustreError: 6881:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 12 previous similar messages [ 228.944261] Lustre: *** cfs_fail_loc=19a, val=0*** [ 228.945951] Lustre: Skipped 45 previous similar messages [ 234.304857] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 17:10:16 (1747257016) [ 234.652853] Lustre: *** cfs_fail_loc=188, val=0*** [ 235.849738] Lustre: *** cfs_fail_loc=189, val=0*** [ 238.709929] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 239.306025] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 17:10:21 (1747257021) [ 241.580104] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 17:10:24 (1747257024) [ 243.918286] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 17:10:26 (1747257026) [ 307.525586] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 17:11:30 (1747257090) [ 312.594970] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 17:11:35 (1747257095) [ 314.971840] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 315.419813] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 17:11:38 (1747257098) [ 317.629122] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 17:11:40 (1747257100) [ 338.506415] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 17:12:01 (1747257121) [ 339.261624] Lustre: *** cfs_fail_loc=725, val=0*** [ 340.240898] Lustre: *** cfs_fail_loc=725, val=0*** [ 342.277825] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 17:12:05 (1747257125) [ 344.959911] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 17:12:07 (1747257127) [ 383.280889] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 17:12:45 (1747257165) [ 394.351399] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 17:12:56 (1747257176) [ 396.577138] Lustre: *** cfs_fail_loc=513, val=0*** [ 396.578827] LustreError: 9754:0:(service.c:2127:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1832131569774784 [ 397.143232] Lustre: Failing over lustre-OST0000 [ 397.161549] Lustre: server umount lustre-OST0000 complete [ 398.190050] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 400.891683] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 400.891740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 400.891979] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 400.903738] LustreError: Skipped 1 previous similar message [ 401.932134] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 401.936407] LustreError: Skipped 2 previous similar messages [ 406.939915] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 406.944616] LustreError: Skipped 2 previous similar messages [ 409.677173] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 409.683245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 409.740543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.757296] mount.lustre (31783) used greatest stack depth: 10280 bytes left [ 410.771677] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 410.787407] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 412.479974] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.123@tcp (at 0@lo) [ 412.480170] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 412.489720] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2519 to 0x0:2561 [ 414.033032] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 414.462777] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 418.053820] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 17:13:20 (1747257200) [ 419.927842] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 17:13:22 (1747257202) [ 422.172075] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 17:13:24 (1747257204) [ 424.529656] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 17:13:27 (1747257207) [ 426.857254] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 17:13:29 (1747257209) [ 428.645170] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 17:13:31 (1747257211) [ 430.543574] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 17:13:33 (1747257213) [ 432.728543] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 17:13:35 (1747257215) [ 435.188882] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 17:13:37 (1747257217) [ 437.993660] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 17:13:40 (1747257220) [ 441.105091] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 17:13:43 (1747257223) [ 441.772282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.777713] Lustre: Skipped 1 previous similar message [ 441.779945] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 441.783338] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.123@tcp (at 0@lo) [ 441.783513] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:35 to 0x280000400:65 [ 441.791272] Lustre: Skipped 1 previous similar message [ 442.083553] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2564 to 0x0:2593 [ 442.407870] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 442.413111] Lustre: Skipped 1 previous similar message [ 442.416789] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:33 [ 442.758900] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2514 to 0x0:2529 [ 448.992487] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 453.413996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 453.419771] Lustre: Skipped 3 previous similar messages [ 453.423017] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 453.426982] Lustre: Skipped 1 previous similar message [ 453.430137] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 453.435597] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.123@tcp (at 0@lo) [ 453.435916] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:35 to 0x280000400:97 [ 453.445253] Lustre: Skipped 3 previous similar messages [ 454.483440] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 454.545649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 455.644891] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 455.704493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 461.141988] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 466.935275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.943020] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 466.947052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 466.954972] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.123@tcp (at 0@lo) [ 466.957337] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3594 to 0x0:3617 [ 468.470795] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 468.538120] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 470.112484] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 470.197487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 477.194735] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 481.977219] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.985873] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 481.989497] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 481.997637] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.201.123@tcp (at 0@lo) [ 481.998141] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:35 to 0x2c0000400:65 [ 483.078045] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 483.138448] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 484.213840] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 484.273350] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 489.619198] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 494.266160] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.271228] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 494.274584] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 494.279116] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.201.123@tcp (at 0@lo) [ 494.279369] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3532 to 0x0:3553 [ 495.322292] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 495.386013] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 496.459748] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 496.518702] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 498.306497] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 17:14:41 (1747257281) [ 500.292451] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 17:14:42 (1747257282) [ 502.228910] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 17:14:44 (1747257284) [ 517.501245] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 17:15:00 (1747257300) [ 521.194350] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 17:15:03 (1747257303) [ 521.835215] Lustre: *** cfs_fail_loc=217, val=0*** [ 522.877027] Lustre: *** cfs_fail_loc=217, val=0*** [ 522.879410] Lustre: Skipped 1 previous similar message [ 525.274056] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 525.826837] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 17:15:08 (1747257308) [ 528.040217] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 17:15:10 (1747257310) [ 530.525739] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 17:15:13 (1747257313) [ 558.846652] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 17:15:41 (1747257341) [ 561.065262] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 17:15:43 (1747257343) [ 563.281614] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 17:15:45 (1747257345) [ 565.550347] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 17:15:48 (1747257348) [ 583.464959] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 17:16:06 (1747257366) [ 594.713281] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 17:16:17 (1747257377) [ 597.286644] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 17:16:19 (1747257379) [ 597.524390] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.23@tcp inode [0x200000407:0xc15:0x0] object 0x0:3817 extent [0-4194303]: client csum 2153cced, server csum 2153ccec [ 599.096377] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 600.259464] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.23@tcp inode [0x200000407:0xc15:0x0] object 0x0:3817 extent [0-4194303], client returned csum f882f449 (type 1), server csum 73c7f967 (type 1) [ 601.122321] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 602.283916] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.23@tcp inode [0x200000407:0xc15:0x0] object 0x0:3817 extent [0-4194303], client returned csum 73b0a54e (type 2), server csum 47f9a5cb (type 2) [ 603.125654] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 604.288436] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.23@tcp inode [0x200000407:0xc15:0x0] object 0x0:3817 extent [0-4194303], client returned csum dc009d65 (type 4), server csum 2153ccec (type 4) [ 605.109902] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 607.024767] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 608.161729] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.23@tcp inode [0x200000407:0xc15:0x0] object 0x0:3817 extent [0-4194303], client returned csum 43da0054 (type 20), server csum 539900d6 (type 20) [ 608.168950] LustreError: Skipped 1 previous similar message [ 608.956413] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 610.872799] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 612.653928] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 614.586556] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 17:16:37 (1747257397) [ 616.696510] Lustre: 9277:0:(tgt_handler.c:1903:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000407:0xc16:0x0]:[0-1048575]-228de2b1-9d3980d9 [ 616.703703] LustreError: dumping log to /tmp/lustre-log.1747257399.9277 [ 616.722734] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.23@tcp inode [0x200000407:0xc16:0x0] object 0x0:3884 extent [0-1048575], client returned csum 228de2b1 (type 4), server csum 9d3980d9 (type 4) [ 616.729810] LustreError: Skipped 2 previous similar messages [ 626.875109] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 17:16:49 (1747257409) [ 627.005800] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.201.23@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 629.309046] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.23@tcp inode [0x200000407:0xc18:0x0] object 0x0:3885 extent [0-4194303], client returned csum f5a99216 (type 4), server csum de2bf73e (type 4) [ 631.166933] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 17:16:53 (1747257413) [ 631.658360] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 631.804254] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.23@tcp inode [0x200000407:0xc19:0x0] object 0x0:3818 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 634.573687] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.23@tcp inode [0x200000407:0xc19:0x0] object 0x0:3818 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 634.579594] LustreError: Skipped 4 previous similar messages [ 641.589933] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.23@tcp inode [0x200000407:0xc19:0x0] object 0x0:3818 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 641.599061] LustreError: Skipped 2 previous similar messages [ 652.689823] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.23@tcp inode [0x200000407:0xc19:0x0] object 0x0:3818 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 652.695714] LustreError: Skipped 3 previous similar messages [ 670.879576] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.23@tcp inode [0x200000407:0xc19:0x0] object 0x0:3818 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 670.888285] LustreError: Skipped 5 previous similar messages [ 677.592714] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 705.698689] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.23@tcp inode [0x200000407:0xc19:0x0] object 0x0:3818 extent [4194304-8388607]: client csum 7d33b9a0, server csum 7d33b99f [ 705.706795] LustreError: Skipped 16 previous similar messages [ 724.235171] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 769.789582] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.23@tcp inode [0x200000407:0xc19:0x0] object 0x0:3818 extent [4194304-8388607]: client csum de2bf73f, server csum de2bf73e [ 769.798632] LustreError: Skipped 27 previous similar messages [ 770.361621] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 817.406091] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 866.430084] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 898.988309] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.23@tcp inode [0x200000407:0xc19:0x0] object 0x0:3818 extent [4194304-8388607]: client csum a30822f0, server csum a30822ef [ 898.993658] LustreError: Skipped 62 previous similar messages [ 911.524945] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 957.596580] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 959.254699] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 17:22:21 (1747257741) [ 959.743652] Lustre: *** cfs_fail_loc=21a, val=0*** [ 961.231708] Lustre: *** cfs_fail_loc=21b, val=0*** [ 964.538425] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 17:22:27 (1747257747) [ 964.806099] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 965.442252] Lustre: Modifying parameter lustre.osc.lustre*.checksums in log params [ 968.090922] Lustre: Disabling parameter lustre.osc.lustre*.checksums in log params [ 970.736275] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 972.272059] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 17:22:34 (1747257754) [ 972.708187] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 973.146085] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 974.693014] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid 40 [ 990.350164] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid in IDLE state after 15 sec [ 991.854741] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid 40 [ 992.238184] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid in FULL state after 0 sec [ 992.670996] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 994.212155] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid 40 [ 1014.933481] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid in IDLE state after 20 sec [ 1016.395462] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid 40 [ 1016.777384] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid in FULL state after 0 sec [ 1017.218984] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1018.808295] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid 40 [ 1040.545705] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid in IDLE state after 21 sec [ 1042.092404] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid 40 [ 1042.463029] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid in FULL state after 0 sec [ 1042.881808] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1044.379794] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid 40 [ 1065.105743] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid in IDLE state after 20 sec [ 1066.522283] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid 40 [ 1066.870173] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid in FULL state after 0 sec [ 1067.248935] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1068.641489] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid 40 [ 1090.394943] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid in IDLE state after 21 sec [ 1091.998461] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid 40 [ 1092.403503] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid in FULL state after 0 sec [ 1092.823240] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1094.326250] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid 40 [ 1115.018865] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid in IDLE state after 20 sec [ 1116.614841] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid 40 [ 1117.016363] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid in FULL state after 0 sec [ 1117.454933] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1118.977205] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid 40 [ 1140.767538] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid in IDLE state after 21 sec [ 1142.336068] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid 40 [ 1142.732497] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880080070800.ost_server_uuid in FULL state after 0 sec [ 1144.458575] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1144.923224] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 17:25:27 (1747257927) [ 1146.586451] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 17:25:29 (1747257929) [ 1147.052017] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1147.445824] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1147.870912] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1148.269661] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1149.928159] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1150.398281] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 17:25:33 (1747257933) [ 1153.543068] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 17:25:36 (1747257936) [ 1155.695942] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ================================================================================= 17:25:38 (1747257938) [ 1167.642681] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 17:25:50 (1747257950) [ 1170.474488] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 17:25:53 (1747257953) [ 1170.809349] Lustre: *** cfs_fail_loc=228, val=0*** [ 1172.525971] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 17:25:55 (1747257955) [ 1172.895163] Lustre: *** cfs_fail_loc=228, val=0*** [ 1174.686922] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 17:25:57 (1747257957) [ 1179.614758] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ===================================================================== 17:26:02 (1747257962) [ 1182.023470] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 17:26:04 (1747257964) [ 1198.133829] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 17:26:20 (1747257980) [ 1201.790730] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 17:26:24 (1747257984) [ 1209.186974] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 17:26:31 (1747257991) [ 1242.703813] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 17:27:05 (1747258025) [ 1250.415557] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 17:27:13 (1747258033) [ 1253.577091] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 17:27:16 (1747258036) [ 1270.049977] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 17:27:32 (1747258052) [ 1275.388938] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 17:27:37 (1747258057) [ 1279.113350] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 17:27:41 (1747258061) [ 1293.308517] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 17:27:55 (1747258075) [ 1296.347691] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 17:27:58 (1747258078) [ 1299.956914] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 17:28:02 (1747258082) [ 1302.458124] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 17:28:05 (1747258085) [ 1307.351734] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 17:28:09 (1747258089) [ 1313.248763] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 17:28:15 (1747258095) [ 1314.107916] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 1314.788761] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 1315.462718] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 1316.216713] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 1318.856747] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 17:28:21 (1747258101) [ 1320.036237] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 1320.703408] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 1321.352375] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 1321.996902] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 1322.690147] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 1325.461628] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 17:28:27 (1747258107) [ 1328.001178] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 17:28:30 (1747258110) [ 1333.944011] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 17:28:36 (1747258116) [ 1336.716570] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 17:28:39 (1747258119) [ 1339.559893] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 17:28:42 (1747258122) [ 1342.075176] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 17:28:44 (1747258124) [ 1344.664936] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 17:28:47 (1747258127) [ 1347.213597] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 17:28:49 (1747258129) [ 1349.203790] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 17:28:51 (1747258131) [ 1351.617164] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 17:28:54 (1747258134) [ 1353.794006] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 17:28:56 (1747258136) [ 1356.183118] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 17:28:58 (1747258138) [ 1414.797756] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 17:29:57 (1747258197) [ 1441.577090] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 17:30:24 (1747258224) [ 1444.038456] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 17:30:26 (1747258226) [ 1740.771790] Lustre: 6851:0:(osd_handler.c:1950:osd_trans_start()) lustre-MDT0000: credits 64189 > trans_max 3200 [ 1740.775315] Lustre: 6851:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 1000/4000/0, destroy: 1/4/0 [ 1740.780195] Lustre: 6851:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 1004/148/0 [ 1740.783378] Lustre: 6851:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 5001/43010/0, punch: 0/0/0, quota 0/0/0 [ 1740.786793] Lustre: 6851:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 1001/17016/0, delete: 2/5/0 [ 1740.790558] Lustre: 6851:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 1740.793382] Pid: 6851, comm: mdt00_000 3.10.0-7.9-debug #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 1740.795990] Call Trace: [ 1740.798168] [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [ 1740.800543] [<0>] libcfs_debug_dumpstack+0x26/0x30 [libcfs] [ 1740.802375] [<0>] osd_trans_start+0x5ba/0x5e0 [osd_ldiskfs] [ 1740.804179] [<0>] top_trans_start+0x763/0xa10 [ptlrpc] [ 1740.805558] [<0>] lod_trans_start+0x34/0x40 [lod] [ 1740.807284] [<0>] mdd_trans_start+0x14/0x20 [mdd] [ 1740.808691] [<0>] mdd_unlink+0x5e3/0xdb0 [mdd] [ 1740.810133] [<0>] mdt_reint_unlink+0xe32/0x1df0 [mdt] [ 1740.811632] [<0>] mdt_reint_rec+0x87/0x240 [mdt] [ 1740.813309] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [ 1740.814861] [<0>] mdt_reint+0x67/0x150 [mdt] [ 1740.816432] [<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc] [ 1740.818080] [<0>] ptlrpc_server_handle_request+0x257/0xc40 [ptlrpc] [ 1740.819830] [<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc] [ 1740.821366] [<0>] kthread+0xe4/0xf0 [ 1740.822474] [<0>] ret_from_fork_nospec_begin+0x7/0x21 [ 1740.823864] [<0>] 0xfffffffffffffffe [ 1740.825414] Lustre: 6851:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 3201 < left 43010, rollback = 7 [ 1742.754858] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 17:35:25 (1747258525) [ 1743.851243] Lustre: lustre-MDD0000: changelog on [ 1744.655662] Lustre: lustre-MDD0001: changelog on [ 1746.727464] Lustre: lustre-MDD0001: changelog off [ 1747.657482] Lustre: lustre-MDD0000: changelog off [ 1748.381627] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 17:35:31 (1747258531) [ 1748.497086] Lustre: lustre-OST0000: Client 7e2ff931-4ad6-4b84-94fd-ff791f4a8ed6 (at 192.168.201.23@tcp) reconnecting [ 1750.045559] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012dc77000.ost_server_uuid 40 [ 1750.428488] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012dc77000.ost_server_uuid in FULL state after 0 sec [ 1752.224070] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 17:35:34 (1747258534) [ 1754.101099] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 17:35:36 (1747258536) [ 1754.522902] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [ 1755.051320] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 17:35:37 (1747258537) [ 1757.007904] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 17:35:39 (1747258539) [ 1759.042656] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 17:35:41 (1747258541) [ 1761.015570] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 17:35:43 (1747258543) [ 1773.120469] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 17:35:55 (1747258555) [ 1775.214118] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 17:35:57 (1747258557) [ 1777.328824] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 17:36:00 (1747258560) [ 1780.821593] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 17:36:03 (1747258563) [ 1783.129694] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [ 1783.721432] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 17:36:06 (1747258566) [ 1856.861345] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 17:37:19 (1747258639) [ 1857.809981] Lustre: *** cfs_fail_loc=147, val=0*** [ 1860.681764] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 17:37:23 (1747258643) [ 1862.918831] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 17:37:25 (1747258645) [ 1865.453727] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 17:37:28 (1747258648) [ 1866.127926] Lustre: *** cfs_fail_loc=217, val=0*** [ 1866.129998] Lustre: Skipped 3 previous similar messages [ 1868.957815] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 1869.481139] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 1870.089605] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 17:37:32 (1747258652) [ 1872.622805] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 17:37:35 (1747258655) [ 1875.195844] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 17:37:37 (1747258657) [ 1875.927319] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1876.942459] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1879.578732] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1881.951623] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1885.956611] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1888.681665] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 17:37:51 (1747258671) [ 1896.956261] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 17:37:59 (1747258679) [ 1897.519956] Lustre: *** cfs_fail_loc=220, val=0*** [ 1897.521806] Lustre: Skipped 3 previous similar messages [ 1900.115461] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 17:38:02 (1747258682) [ 1910.347404] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 17:38:12 (1747258692) [ 1912.403754] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 17:38:15 (1747258695) [ 1914.407721] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 17:38:17 (1747258697) [ 1919.129272] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 17:38:21 (1747258701) [ 1921.173892] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 17:38:23 (1747258703) [ 1923.220242] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 17:38:25 (1747258705) [ 1925.241002] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ========================================================== 17:38:27 (1747258707) [ 1926.781087] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [ 1930.789484] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 17:38:33 (1747258713) [ 1933.500066] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 17:38:36 (1747258716) [ 1936.125299] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 17:38:38 (1747258718) [ 1938.589203] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 17:38:41 (1747258721) [ 1940.903636] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 17:38:43 (1747258723) [ 1950.436440] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 17:38:53 (1747258733) [ 1960.084173] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 17:39:02 (1747258742) [ 2022.525634] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 17:40:05 (1747258805) [ 2024.402999] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 17:40:07 (1747258807) [ 2025.612167] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 2026.352491] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 2034.147087] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 3 sec [ 2035.129880] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 0 sec [ 2102.516106] Lustre: DEBUG MARKER: ls -l 10000 files without statahead: 39 sec [ 2107.989934] Lustre: DEBUG MARKER: ls -l 10000 files with statahead: 4 sec [ 2108.451904] Lustre: DEBUG MARKER: ls -l done [ 2128.260378] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity/: 19 seconds [ 2128.840032] Lustre: DEBUG MARKER: rm done [ 2130.808682] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 17:41:53 (1747258913) [ 2131.253144] Lustre: DEBUG MARKER: SKIP: sanity test_123ab Test must be statx() syscall supported [ 2131.797286] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 17:41:54 (1747258914) [ 2132.255406] Lustre: DEBUG MARKER: SKIP: sanity test_123ac Test must be statx() syscall supported [ 2132.777142] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 17:41:55 (1747258915) [ 2140.267098] Lustre: DEBUG MARKER: ls done [ 2143.872666] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 17:42:06 (1747258926) [ 2146.564063] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 17:42:09 (1747258929) [ 2147.093942] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 2155.051101] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800aa9a3800 [ 2155.568679] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800aa9a3800 [ 2156.044726] Lustre: DEBUG MARKER: LRU=1004 [ 2156.526260] Lustre: DEBUG MARKER: LIMIT=46624 [ 2156.980802] Lustre: DEBUG MARKER: LVF=5572500 [ 2157.411828] Lustre: DEBUG MARKER: OLD_LVF=100 [ 2157.858829] Lustre: DEBUG MARKER: Sleep 50 sec [ 2208.387675] Lustre: DEBUG MARKER: Dropped 402 locks in 50s [ 2208.806554] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 2214.539942] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 17:43:17 (1747258997) [ 2236.358749] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 2263.019816] Lustre: DEBUG MARKER: ls -la time: 26 seconds [ 2263.422191] Lustre: DEBUG MARKER: lru_size = 400 [ 2294.779492] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 2301.260987] Lustre: DEBUG MARKER: ls -la time: 5 seconds [ 2301.639792] Lustre: DEBUG MARKER: lru_size = 4052 [ 2301.996226] Lustre: DEBUG MARKER: ls -la is 80% faster with lru resize enabled [ 2312.217870] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 17:44:54 (1747259094) [ 2334.466650] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize diasbaled ========================================================== 17:45:17 (1747259117) [ 2356.587525] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 17:45:39 (1747259139) [ 2358.276177] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 17:45:41 (1747259141) [ 2359.903332] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 17:45:42 (1747259142) [ 2361.694229] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 17:45:44 (1747259144) [ 2363.386912] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [ 2367.531808] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 17:45:50 (1747259150) [ 2369.165469] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 17:45:51 (1747259151) [ 2373.158292] Lustre: 13653:0:(osd_handler.c:586:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 1806, FID: [0x20000040b:0x23d8:0x0]) is approaching max size limit [ 2376.349022] Lustre: 6853:0:(osd_handler.c:582:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 1806, FID: [0x20000040b:0x23d8:0x0]) has reached max size limit [ 2383.881966] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 17:46:06 (1747259166) [ 2385.573885] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 17:46:08 (1747259168) [ 2387.228320] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 17:46:09 (1747259169) [ 2388.902564] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 17:46:11 (1747259171) [ 2389.262277] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [ 2389.698776] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 17:46:12 (1747259172) [ 2397.261745] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 17:46:20 (1747259180) [ 2398.882224] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 17:46:21 (1747259181) [ 2401.885408] Lustre: 13653:0:(osd_handler.c:1950:osd_trans_start()) lustre-MDT0000: credits 12989 > trans_max 3200 [ 2401.887876] Lustre: 13653:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 200/800/0, destroy: 1/4/0 [ 2401.890322] Lustre: 13653:0:(osd_handler.c:1879:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2401.892996] Lustre: 13653:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 204/148/0 [ 2401.895224] Lustre: 13653:0:(osd_handler.c:1886:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2401.898522] Lustre: 13653:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 1001/8610/0, punch: 0/0/0, quota 0/0/0 [ 2401.901242] Lustre: 13653:0:(osd_handler.c:1896:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2401.904069] Lustre: 13653:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 201/3416/0, delete: 2/5/0 [ 2401.906335] Lustre: 13653:0:(osd_handler.c:1903:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2401.909072] Lustre: 13653:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 2401.911620] Lustre: 13653:0:(osd_handler.c:1910:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2401.914017] Pid: 13653, comm: mdt00_006 3.10.0-7.9-debug #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 2401.916467] Call Trace: [ 2401.917477] [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [ 2401.918715] [<0>] libcfs_debug_dumpstack+0x26/0x30 [libcfs] [ 2401.920297] [<0>] osd_trans_start+0x5ba/0x5e0 [osd_ldiskfs] [ 2401.922604] [<0>] top_trans_start+0x763/0xa10 [ptlrpc] [ 2401.924016] [<0>] lod_trans_start+0x34/0x40 [lod] [ 2401.925341] [<0>] mdd_trans_start+0x14/0x20 [mdd] [ 2401.926896] [<0>] mdd_unlink+0x5e3/0xdb0 [mdd] [ 2401.927976] [<0>] mdt_reint_unlink+0xe32/0x1df0 [mdt] [ 2401.929602] [<0>] mdt_reint_rec+0x87/0x240 [mdt] [ 2401.930712] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [ 2401.931812] [<0>] mdt_reint+0x67/0x150 [mdt] [ 2401.933168] [<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc] [ 2401.934700] [<0>] ptlrpc_server_handle_request+0x257/0xc40 [ptlrpc] [ 2401.936481] [<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc] [ 2401.937787] [<0>] kthread+0xe4/0xf0 [ 2401.938748] [<0>] ret_from_fork_nospec_begin+0x7/0x21 [ 2401.940218] [<0>] 0xfffffffffffffffe [ 2401.941403] Lustre: 13653:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 3203 < left 8610, rollback = 7 [ 2401.943775] Lustre: 13653:0:(osd_internal.h:1333:osd_trans_exec_op()) Skipped 4000 previous similar messages [ 2402.434278] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 17:46:25 (1747259185) [ 2404.006087] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 17:46:26 (1747259186) [ 2405.661540] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 17:46:28 (1747259188) [ 2407.189784] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 17:46:29 (1747259189) [ 2408.860124] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 17:46:31 (1747259191) [ 2410.470892] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 17:46:33 (1747259193) [ 2415.248755] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 17:46:37 (1747259197) [ 2418.213541] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 17:46:40 (1747259200) [ 2434.918970] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 17:46:57 (1747259217) [ 2441.554509] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 17:47:04 (1747259224) [ 2444.408642] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 17:47:07 (1747259227) [ 2451.003883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.004152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2451.011385] Lustre: Skipped 3 previous similar messages [ 2453.600504] Lustre: server umount lustre-MDT0000 complete [ 2455.792922] LustreError: 6839:0:(ldlm_lockd.c:2527:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1747259238 with bad export cookie 3327610705184570223 [ 2455.793881] LustreError: 166-1: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2455.800686] LustreError: 6839:0:(ldlm_lockd.c:2527:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2455.906554] Lustre: server umount lustre-MDT0001 complete [ 2464.071503] Lustre: 9819:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1747259240/real 1747259240] req@ffff88008c4404c0 x1832131578993280/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1747259246 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2464.102216] Lustre: server umount lustre-OST0000 complete [ 2472.414467] Lustre: 10419:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1747259248/real 1747259248] req@ffff88008c442f80 x1832131578993664/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1747259254 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2472.506243] Lustre: server umount lustre-OST0001 complete [ 2474.201934] device-mapper: core: cleaned up [ 2476.837924] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2477.339973] Key type lgssc unregistered [ 2477.401613] LNet: 11352:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2477.403913] LNet: Removed LNI 192.168.201.123@tcp [ 2481.993555] alg: No test for adler32 (adler32-zlib) [ 2482.768555] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2482.972052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2483.038793] Lustre: Lustre: Build Version: 2.15.6_28_g2a10569 [ 2483.090005] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2483.092572] LNet: Accept secure, port 988 [ 2484.622432] Key type lgssc registered [ 2484.840572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2488.704037] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2489.830044] device-mapper: uevent: version 1.0.3 [ 2489.832089] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 2491.713389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.833832] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.851417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.870599] mount.lustre (13706) used greatest stack depth: 9728 bytes left [ 2493.833277] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2496.872400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.965841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2497.909582] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2500.988556] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2500.992702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2501.082499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2501.985943] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2502.079988] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.081861] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11069 to 0x280000400:11105 [ 2504.955501] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2504.962609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2505.010545] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2505.021808] mount.lustre (17124) used greatest stack depth: 9568 bytes left [ 2505.877879] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2508.011359] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:10903 to 0x2c0000400:10945 [ 2508.012345] Lustre: lustre-OST0001: deleting orphan objects from 0x0:16551 to 0x0:16577 [ 2508.013980] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16713 to 0x0:16737 [ 2513.485296] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2514.412785] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2521.130399] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 17:48:23 (1747259303) [ 2537.019689] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2537.023654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.028002] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2538.059885] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.059942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2538.066898] Lustre: Skipped 2 previous similar messages [ 2540.924818] Lustre: server umount lustre-MDT0000 complete [ 2543.067686] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.070938] LustreError: Skipped 3 previous similar messages [ 2543.078757] LustreError: 13713:0:(ldlm_lockd.c:2527:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1747259325 with bad export cookie 18118750447567949815 [ 2543.079370] LustreError: 166-1: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2543.085488] LustreError: 13713:0:(ldlm_lockd.c:2527:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2543.165692] Lustre: server umount lustre-MDT0001 complete [ 2551.352813] Lustre: 22163:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1747259327/real 1747259327] req@ffff88013731ed40 x1832134131158720/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1747259333 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2551.369733] Lustre: server umount lustre-OST0000 complete [ 2559.690446] Lustre: 22762:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1747259336/real 1747259336] req@ffff8800627b50c0 x1832134131159168/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1747259342 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2559.750398] Lustre: server umount lustre-OST0001 complete [ 2561.506037] device-mapper: core: cleaned up [ 2564.159364] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2564.669689] Key type lgssc unregistered [ 2564.728648] LNet: 23695:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2564.732812] LNet: Removed LNI 192.168.201.123@tcp [ 2569.142700] alg: No test for adler32 (adler32-zlib) [ 2569.915986] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2570.033068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2570.121771] Lustre: Lustre: Build Version: 2.15.6_28_g2a10569 [ 2570.165000] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2570.166958] LNet: Accept secure, port 988 [ 2571.692470] Key type lgssc registered [ 2571.905478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2575.220250] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2576.307921] device-mapper: uevent: version 1.0.3 [ 2576.309276] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 2578.201906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.319528] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.335151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2580.204881] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2583.145709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.239020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2584.165171] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2587.087057] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2587.090592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2587.164677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2588.066379] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2588.163285] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.164671] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11069 to 0x280000400:11137 [ 2591.212754] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2591.216296] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2591.260335] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2591.795670] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:10903 to 0x2c0000400:10977 [ 2592.283095] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2593.794314] Lustre: lustre-OST0001: deleting orphan objects from 0x0:16551 to 0x0:16609 [ 2593.794519] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16713 to 0x0:16769 [ 2600.107618] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2601.085583] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2607.777959] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 17:49:50 (1747259390) [ 2793.656664] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 17:52:56 (1747259576) [ 2796.632377] Lustre: *** cfs_fail_loc=327, val=0*** [ 2810.799957] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 17:53:13 (1747259593) [ 2812.473401] Lustre: *** cfs_fail_loc=328, val=0*** [ 2812.475104] Lustre: Skipped 512 previous similar messages [ 2813.478105] Lustre: *** cfs_fail_loc=328, val=0*** [ 2815.976613] Lustre: *** cfs_fail_loc=328, val=0*** [ 2815.977990] Lustre: Skipped 252 previous similar messages [ 2820.210284] Lustre: *** cfs_fail_loc=328, val=0*** [ 2820.211693] Lustre: Skipped 125 previous similar messages [ 2828.353478] Lustre: *** cfs_fail_loc=328, val=0*** [ 2828.355311] Lustre: Skipped 65 previous similar messages [ 2832.626944] LustreError: 18237:0:(ldlm_resource.c:133:seq_watermark_write()) Failed to set lock_reclaim_threshold_mb, rc = -22. [ 2836.388204] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [ 2836.872305] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [ 2837.354644] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 17:53:40 (1747259620) [ 2843.383293] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 17:53:46 (1747259626) [ 2856.722688] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 17:53:59 (1747259639) [ 2866.647999] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 17:54:09 (1747259649) [ 2876.455071] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 17:54:19 (1747259659) [ 2887.277077] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 17:54:30 (1747259670) [ 2890.975439] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 17:54:33 (1747259673) [ 2909.265041] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 17:54:52 (1747259692) [ 2922.323294] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 17:55:05 (1747259705) [ 2931.897167] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 17:55:14 (1747259714) [ 2935.984427] bash (27330): drop_caches: 1 [ 2938.916284] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 17:55:21 (1747259721) [ 2940.423977] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 17:55:23 (1747259723) [ 2941.988979] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 17:55:24 (1747259724) [ 2943.545882] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 17:55:26 (1747259726) [ 2945.148356] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 17:55:27 (1747259727) [ 2945.326919] LustreError: 28708:0:(fld_handler.c:263:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 2947.080749] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 17:55:29 (1747259729) [ 2947.267953] LustreError: 28708:0:(fld_handler.c:263:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 2947.270115] LustreError: 28708:0:(fld_handler.c:263:fld_server_lookup()) Skipped 3 previous similar messages [ 2949.022082] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 17:55:31 (1747259731) [ 2950.554193] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 17:55:33 (1747259733) [ 2952.363553] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 17:55:35 (1747259735) [ 2953.952863] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 17:55:36 (1747259736) [ 2955.752633] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 17:55:38 (1747259738) [ 3040.133295] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 17:57:02 (1747259822) [ 3043.550488] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 17:57:06 (1747259826) [ 3047.122248] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 17:57:09 (1747259829) [ 3050.783099] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 17:57:13 (1747259833) [ 3054.489183] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 17:57:17 (1747259837) [ 3067.380198] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 17:57:30 (1747259850) [ 3081.895924] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 17:57:44 (1747259864) [ 3092.099954] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 17:57:54 (1747259874) [ 3107.587958] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 17:58:10 (1747259890) [ 3110.046669] Lustre: DEBUG MARKER: Turn on read and write cache [ 3111.174867] Lustre: DEBUG MARKER: Write data and read it back. [ 3111.574525] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 3112.748849] Lustre: DEBUG MARKER: cache hits: before: 65581, after: 65584 [ 3113.134343] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 3113.903810] Lustre: DEBUG MARKER: cache hits:: before: 65584, after: 65587 [ 3114.289780] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [ 3115.382465] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 3116.470466] Lustre: DEBUG MARKER: cache hits:: before: 65587, after: 65590 [ 3116.885497] Lustre: DEBUG MARKER: Write data and read it back. [ 3117.281794] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 3118.375667] Lustre: DEBUG MARKER: cache hits:: before: 65590, after: 65593 [ 3118.775115] Lustre: DEBUG MARKER: Turn off read and write cache [ 3119.852892] Lustre: DEBUG MARKER: Write data and read it back [ 3120.229339] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 3121.344018] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65593 [ 3121.736969] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [ 3122.876117] Lustre: DEBUG MARKER: Write data and read it back [ 3123.272951] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 3124.373761] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65593 [ 3124.749468] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 3125.803867] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65596 [ 3127.963729] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 17:58:30 (1747259910) [ 3128.723518] Lustre: lustre-MDD0000: changelog on [ 3129.467856] Lustre: lustre-MDD0001: changelog on [ 3133.459994] Lustre: Failing over lustre-MDT0000 [ 3134.107798] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3134.110645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.114494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3134.667757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.667937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3134.667939] Lustre: Skipped 1 previous similar message [ 3134.674808] Lustre: Skipped 2 previous similar messages [ 3138.626916] Lustre: server umount lustre-MDT0000 complete [ 3139.165938] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.859648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.891581] LustreError: 166-1: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3140.891612] LustreError: 11-0: MGC192.168.201.123@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 3140.897134] Lustre: Evicted from MGS (at 192.168.201.123@tcp) after server handle changed from 0xdc2b222c1b7b15e9 to 0xdc2b222c1b83cd8b [ 3140.901337] Lustre: MGC192.168.201.123@tcp: Connection restored to 192.168.201.123@tcp (at 0@lo) [ 3140.956420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.963477] Lustre: lustre-MDD0000: changelog on [ 3140.971757] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3141.920118] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3144.174001] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3145.964702] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 3145.992820] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3146.009310] Lustre: lustre-OST0001: deleting orphan objects from 0x0:17438 to 0x0:17473 [ 3146.009530] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17594 to 0x0:17633 [ 3146.226537] Lustre: lustre-MDD0000: changelog off [ 3150.059193] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 17:58:52 (1747259932) [ 3150.823722] Lustre: lustre-MDD0000: changelog on [ 3153.571787] Lustre: lustre-MDD0001: changelog off [ 3153.573097] Lustre: Skipped 1 previous similar message [ 3154.928032] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 17:58:57 (1747259937) [ 3155.709459] Lustre: lustre-MDD0000: changelog on [ 3155.710820] Lustre: Skipped 1 previous similar message [ 3158.968602] Lustre: lustre-MDD0001: changelog off [ 3158.970784] Lustre: Skipped 1 previous similar message [ 3160.501384] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 17:59:03 (1747259943) [ 3164.197005] Lustre: lustre-MDD0001: changelog off [ 3164.198459] Lustre: Skipped 1 previous similar message [ 3165.681524] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 17:59:08 (1747259948) [ 3166.593650] Lustre: lustre-MDD0000: changelog on [ 3166.596564] Lustre: Skipped 3 previous similar messages [ 3169.688814] Lustre: lustre-MDD0001: changelog off [ 3169.689891] Lustre: Skipped 1 previous similar message [ 3171.100333] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 17:59:13 (1747259953) [ 3174.465496] Lustre: DEBUG MARKER: 1747259957: creating first files [ 3189.930234] Lustre: *** cfs_fail_loc=1313, val=0*** [ 3189.932451] Lustre: 27226:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 3189.937450] Lustre: 19143:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl7 idle for 17s with 4 unprocessed records [ 3195.589373] Lustre: lustre-MDD0001: changelog off [ 3195.590634] Lustre: Skipped 1 previous similar message [ 3197.054229] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 17:59:39 (1747259979) [ 3197.859075] Lustre: lustre-MDD0000: changelog on [ 3197.860180] Lustre: Skipped 3 previous similar messages [ 3203.675390] Lustre: 26070:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 3203.678448] Lustre: 26070:0:(mdd_dir.c:895:mdd_changelog_store()) Skipped 1 previous similar message [ 3203.681664] Lustre: 22423:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl9 idle for 5s with 4 unprocessed records [ 3203.686938] Lustre: 22423:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 3210.309754] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 17:59:53 (1747259993) [ 3228.187189] Lustre: *** cfs_fail_loc=1316, val=0*** [ 3228.189306] Lustre: 27226:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0001: simulate starting changelog garbage collection [ 3228.193887] Lustre: 27226:0:(mdd_dir.c:895:mdd_changelog_store()) Skipped 1 previous similar message [ 3228.196647] Lustre: 25731:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0001: force deregister of changelog user cl11 idle for 15s with 3 unprocessed records [ 3228.203145] Lustre: 25731:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) Skipped 2 previous similar messages [ 3228.703588] Lustre: Failing over lustre-MDT0001 [ 3228.789204] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_disconnect to node 0@lo failed: rc = -107 [ 3229.310855] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.23@tcp (stopping) [ 3229.314103] Lustre: Skipped 2 previous similar messages [ 3231.099567] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3231.102906] Lustre: Skipped 3 previous similar messages [ 3231.253851] Lustre: server umount lustre-MDT0001 complete [ 3233.474704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3233.505243] LustreError: 166-1: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3233.511122] Lustre: Evicted from MGS (at 192.168.201.123@tcp) after server handle changed from 0xdc2b222c1b83cd8b to 0xdc2b222c1b83de86 [ 3233.514545] Lustre: MGC192.168.201.123@tcp: Connection restored to (at 0@lo) [ 3233.517465] Lustre: Skipped 3 previous similar messages [ 3233.588448] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3233.597477] LustreError: Skipped 8 previous similar messages [ 3233.603953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3233.613277] Lustre: lustre-MDD0000: changelog on [ 3233.614355] Lustre: Skipped 3 previous similar messages [ 3233.615611] Lustre: 27483:0:(mdd_device.c:618:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 24 to index 25, being cleared now [ 3233.620846] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3234.318027] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3234.508902] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3236.730713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.822589] Lustre: 28462:0:(mdd_device.c:618:mdd_changelog_llog_init()) lustre-MDD0001 : orphan changelog records found, starting from index 22 to index 23, being cleared now [ 3236.829296] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 3237.715019] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3238.398312] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3238.858320] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3238.858370] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 3238.879971] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11145 to 0x280000400:11169 [ 3238.880048] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:10985 to 0x2c0000400:11009 [ 3238.880399] Lustre: lustre-OST0001: deleting orphan objects from 0x0:17438 to 0x0:17505 [ 3238.880815] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17635 to 0x0:17665 [ 3243.135486] Lustre: lustre-MDD0001: changelog off [ 3243.136564] Lustre: Skipped 3 previous similar messages [ 3243.820743] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 3243.822438] Lustre: Skipped 1 previous similar message [ 3244.514148] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 18:00:27 (1747260027) [ 3247.106915] LustreError: 31071:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 1315 sleeping [ 3249.326958] LustreError: 31241:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 1315 waking [ 3249.328810] LustreError: 31071:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 1315 awake: rc=2780 [ 3250.033829] LustreError: 31578:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 1315 waking [ 3250.036708] LustreError: 31578:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message [ 3255.515782] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 18:00:38 (1747260038) [ 3261.088899] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 18:00:43 (1747260043) [ 3263.060737] LustreError: 27493:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 15d sleeping for 3000ms [ 3266.064460] LustreError: 27493:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 15d awake [ 3273.265684] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 18:00:55 (1747260055) [ 3278.564315] Lustre: lustre-MDD0001: changelog off [ 3278.565416] Lustre: Skipped 9 previous similar messages [ 3279.956210] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 18:01:02 (1747260062) [ 3283.634683] LustreError: 31098:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 15f sleeping [ 3285.638374] LustreError: 27751:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15f waking [ 3285.640349] LustreError: 31098:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 15f awake: rc=2996 [ 3290.206897] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 18:01:12 (1747260072) [ 3564.654178] LustreError: 27494:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 16c sleeping [ 3566.655633] LustreError: 1105:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 16c waking [ 3566.658603] LustreError: 27494:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 16c awake: rc=2998 [ 3568.652579] Lustre: lustre-MDD0001: changelog off [ 3568.653668] Lustre: Skipped 3 previous similar messages [ 3570.180067] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 18:05:52 (1747260352) [ 3570.947573] Lustre: lustre-MDD0000: changelog on [ 3570.949304] Lustre: Skipped 15 previous similar messages [ 3571.968229] LustreError: 9181:0:(mdd_device.c:1704:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [ 3572.233848] Lustre: 9245:0:(mdd_device.c:1721:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [ 3572.484160] LustreError: 9309:0:(mdd_device.c:1713:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [ 3580.958735] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 18:06:03 (1747260363) [ 3583.816299] Lustre: Failing over lustre-MDT0000 [ 3583.818257] Lustre: Skipped 1 previous similar message [ 3583.886971] Lustre: server umount lustre-MDT0000 complete [ 3583.889448] Lustre: Skipped 1 previous similar message [ 3584.227735] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3584.234601] LustreError: Skipped 1 previous similar message [ 3584.363875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3584.368085] Lustre: Skipped 3 previous similar messages [ 3586.691930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.743752] LustreError: 11-0: MGC192.168.201.123@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 3586.743842] LustreError: 166-1: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3586.744122] Lustre: Evicted from MGS (at 192.168.201.123@tcp) after server handle changed from 0xdc2b222c1b83de86 to 0xdc2b222c1bc754f2 [ 3586.755356] Lustre: MGC192.168.201.123@tcp: Connection restored to (at 0@lo) [ 3586.759802] Lustre: Skipped 3 previous similar messages [ 3586.827536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3586.830513] Lustre: Skipped 1 previous similar message [ 3586.915211] Lustre: 12288:0:(mdd_device.c:618:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 90148 to index 18446744073709551615, being cleared now [ 3586.928271] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3588.222958] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3589.230717] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3591.847591] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3591.854130] Lustre: Skipped 1 previous similar message [ 3591.880887] Lustre: lustre-OST0001: deleting orphan objects from 0x0:17507 to 0x0:17537 [ 3591.880934] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17667 to 0x0:17697 [ 3595.571508] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 18:06:18 (1747260378) [ 3598.982434] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records anaconda-ks.cfg stress.sh time ========================================================== 18:06:21 (1747260381) [ 3605.706812] Lustre: 27494:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 3605.712077] Lustre: 27494:0:(mdd_dir.c:895:mdd_changelog_store()) Skipped 1 previous similar message [ 3605.717749] Lustre: 14930:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl2 idle for 864006s with 500000004 unprocessed records [ 3610.511932] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 18:06:33 (1747260393) [ 3620.910608] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 18:06:43 (1747260403) [ 3628.965048] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 18:06:51 (1747260411) [ 3634.215884] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 18:06:56 (1747260416) [ 3641.099986] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 18:07:03 (1747260423) [ 3643.008691] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 18:07:05 (1747260425) [ 3644.849186] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 18:07:07 (1747260427) [ 3657.165021] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 18:07:19 (1747260439) [ 3663.103233] Lustre: Failing over lustre-OST0000 [ 3663.115271] Lustre: server umount lustre-OST0000 complete [ 3664.355741] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3664.359117] LustreError: Skipped 8 previous similar messages [ 3666.955794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3666.959099] Lustre: Skipped 1 previous similar message [ 3667.854837] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3667.859295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3667.908448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3667.914336] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3668.894590] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3668.990675] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3668.994749] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3668.994753] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.123@tcp (at 0@lo) [ 3668.994754] Lustre: Skipped 4 previous similar messages [ 3668.996198] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11175 to 0x280000400:11201 [ 3668.997851] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17701 to 0x0:17729 [ 3669.915110] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 18:07:32 (1747260452) [ 3692.959456] Lustre: Failing over lustre-OST0000 [ 3692.970491] Lustre: server umount lustre-OST0000 complete [ 3694.043546] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3694.043652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.043878] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3694.043880] LustreError: Skipped 2 previous similar messages [ 3694.055021] LustreError: Skipped 1 previous similar message [ 3695.062270] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3695.066232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3695.106723] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3696.001735] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3696.633966] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3696.803676] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3696.805265] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11175 to 0x280000400:11233 [ 3696.805786] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17731 to 0x0:17761 [ 3696.963438] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 18:07:59 (1747260479) [ 3706.698846] Lustre: Failing over lustre-OST0000 [ 3706.715846] Lustre: server umount lustre-OST0000 complete [ 3706.827716] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3709.040233] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3709.044372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3710.069617] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3710.310623] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3710.310704] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.123@tcp (at 0@lo) [ 3710.310706] Lustre: Skipped 3 previous similar messages [ 3710.313139] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17829 to 0x0:17857 [ 3710.314342] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11295 to 0x280000400:11329 [ 3711.114826] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 18:08:13 (1747260493) [ 3731.364227] Lustre: Failing over lustre-OST0000 [ 3731.372465] Lustre: server umount lustre-OST0000 complete [ 3733.486615] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3733.489920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3733.525191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3733.529037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3733.529039] Lustre: Skipped 2 previous similar messages [ 3733.530282] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3733.530284] Lustre: Skipped 1 previous similar message [ 3733.539169] Lustre: Skipped 3 previous similar messages [ 3733.541394] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3733.543832] Lustre: Skipped 1 previous similar message [ 3734.436237] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3734.471044] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3734.472688] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11295 to 0x280000400:11361 [ 3734.473205] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17859 to 0x0:17889 [ 3735.392845] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 18:08:38 (1747260518) [ 3748.302302] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3748.305684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3749.277182] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3749.612394] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17891 to 0x0:17921 [ 3749.612735] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11363 to 0x280000400:11393 [ 3750.270163] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 18:08:52 (1747260532) [ 3758.364111] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3758.368568] LustreError: Skipped 5 previous similar messages [ 3760.676180] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3760.680424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3761.599918] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3762.198287] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17891 to 0x0:17953 [ 3762.198714] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11363 to 0x280000400:11425 [ 3762.533911] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 18:09:05 (1747260545) [ 3762.933136] Lustre: DEBUG MARKER: creating a 10 Mb file [ 3763.386956] Lustre: DEBUG MARKER: starting reads [ 3763.787397] Lustre: DEBUG MARKER: truncating the file [ 3764.225204] Lustre: DEBUG MARKER: killing dd [ 3764.617738] Lustre: DEBUG MARKER: removing the temporary file [ 3766.270181] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log =============================================================================== 18:09:08 (1747260548) [ 3768.078180] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 18:09:10 (1747260550) [ 3772.864478] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 18:09:15 (1747260555) [ 3773.314606] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [ 3773.790571] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 18:09:16 (1747260556) [ 3774.513431] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_module obdecho/obdecho [ 3779.140304] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 18:09:21 (1747260561) [ 3779.888148] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_module obdecho/obdecho [ 3779.934363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3785.425495] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 18:09:28 (1747260568) [ 3799.655351] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ========================================================================== 18:09:42 (1747260582) [ 3808.918127] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 18:09:51 (1747260591) [ 3809.213029] Lustre: *** cfs_fail_loc=148, val=0*** [ 3811.242677] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 18:09:53 (1747260593) [ 3813.836689] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 18:09:56 (1747260596) [ 3815.662744] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 18:09:58 (1747260598) [ 3823.065877] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 18:10:05 (1747260605) [ 3826.826961] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 18:10:09 (1747260609) [ 3830.264539] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 18:10:13 (1747260613) [ 3831.849641] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 18:10:14 (1747260614) [ 3834.571493] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 18:10:17 (1747260617) [ 3840.315514] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 18:10:23 (1747260623) [ 3842.306711] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 18:10:25 (1747260625) [ 3844.004575] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 18:10:26 (1747260626) [ 3859.232727] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 18:10:41 (1747260641) [ 3860.984136] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 18:10:43 (1747260643) [ 3862.641895] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 18:10:45 (1747260645) [ 3864.454225] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 18:10:47 (1747260647) [ 3866.254551] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 18:10:48 (1747260648) [ 3867.987661] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 18:10:50 (1747260650) [ 3869.670666] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 18:10:52 (1747260652) [ 3871.322269] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 18:10:54 (1747260654) [ 3872.961929] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 18:10:55 (1747260655) [ 3876.034767] Lustre: lustre-MDD0000: changelog on [ 3876.036000] Lustre: Skipped 10 previous similar messages [ 3877.502065] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [ 3877.916989] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.15340 [ 3878.592197] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [ 3879.000963] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.11681 [ 3879.650080] Lustre: DEBUG MARKER: Test: lfs mkdir -i 1 /mnt/lustre/d205a.sanity.remote [ 3880.060216] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.19450 [ 3880.717825] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [ 3881.096229] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.11893 [ 3881.748930] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [ 3882.149117] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.13582 [ 3882.791798] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [ 3883.183096] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.15193 [ 3883.829167] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3884.225658] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.25139 [ 3885.165473] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [ 3885.596811] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.4830 [ 3886.340943] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [ 3886.755020] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.26663 [ 3887.442925] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [ 3887.864729] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.23759 [ 3888.809642] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [ 3889.229959] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.21170 [ 3889.908282] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [ 3890.311555] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.17768 [ 3893.786992] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3894.218066] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg123-client.v [ 3894.944230] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3895.409827] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg123-client.E [ 3896.128071] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 3896.552644] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.oleg123-client.v [ 3898.906311] Lustre: lustre-MDD0001: changelog off [ 3898.908417] Lustre: Skipped 11 previous similar messages [ 3900.643311] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 18:11:23 (1747260683) [ 3903.643518] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 18:11:26 (1747260686) [ 3905.342883] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 18:11:28 (1747260688) [ 3907.029516] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 18:11:29 (1747260689) [ 3908.723703] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 18:11:31 (1747260691) [ 3910.453168] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 18:11:33 (1747260693) [ 3917.050647] Lustre: Failing over lustre-MDT0000 [ 3917.051878] Lustre: Skipped 2 previous similar messages [ 3917.118299] Lustre: server umount lustre-MDT0000 complete [ 3917.119578] Lustre: Skipped 2 previous similar messages [ 3917.387644] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3917.390676] LustreError: Skipped 1 previous similar message [ 3917.392306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3917.396424] Lustre: Skipped 5 previous similar messages [ 3917.398121] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3917.402351] LustreError: Skipped 1 previous similar message [ 3927.979380] Lustre: 24214:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1747260703/real 1747260703] req@ffff88006696a600 x1832134223955904/t0(0) o400->MGC192.168.201.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1747260710 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 3927.986573] LustreError: 166-1: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3929.564521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3934.004425] Lustre: Evicted from MGS (at 192.168.201.123@tcp) after server handle changed from 0xdc2b222c1bc754f2 to 0xdc2b222c1bdbefa3 [ 3934.010181] LustreError: 22030:0:(ldlm_resource.c:1126:ldlm_resource_complain()) MGC192.168.201.123@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (ffff880130912b00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3934.017644] Lustre: MGC192.168.201.123@tcp: Connection restored to 192.168.201.123@tcp (at 0@lo) [ 3934.021837] Lustre: Skipped 7 previous similar messages [ 3934.086273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.089939] Lustre: Skipped 2 previous similar messages [ 3934.100300] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3934.103396] Lustre: Skipped 2 previous similar messages [ 3935.095286] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3935.538634] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3935.542866] Lustre: Skipped 2 previous similar messages [ 3939.105187] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3939.112274] Lustre: Skipped 2 previous similar messages [ 3939.130395] Lustre: lustre-OST0000: deleting orphan objects from 0x0:21470 to 0x0:21505 [ 3939.130401] Lustre: lustre-OST0001: deleting orphan objects from 0x0:21049 to 0x0:21089 [ 3940.522152] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3940.923727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3956.123474] Lustre: 24213:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1747260731/real 1747260731] req@ffff88012ba584c0 x1832134223967936/t0(0) o400->MGC192.168.201.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1747260738 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 3956.132612] LustreError: 166-1: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3957.284976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3962.148571] Lustre: Evicted from MGS (at 192.168.201.123@tcp) after server handle changed from 0xdc2b222c1bdbefa3 to 0xdc2b222c1bdbf488 [ 3963.222518] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3967.247137] Lustre: lustre-OST0000: deleting orphan objects from 0x0:21470 to 0x0:21537 [ 3967.249582] Lustre: lustre-OST0001: deleting orphan objects from 0x0:21049 to 0x0:21121 [ 3968.650074] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3969.051637] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3972.130154] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 18:12:34 (1747260754) [ 3980.783868] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 18:12:43 (1747260763) [ 3984.753520] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 18:12:47 (1747260767) [ 3987.012656] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 18:12:49 (1747260769) [ 3998.925968] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 18:13:01 (1747260781) [ 4003.812230] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 18:13:06 (1747260786) [ 4005.758926] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 18:13:08 (1747260788) [ 4011.335972] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') ========================================================== 18:13:14 (1747260794) [ 4013.480802] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 18:13:16 (1747260796) [ 4013.911674] Lustre: DEBUG MARKER: creating a 10 Mb file [ 4014.374595] Lustre: DEBUG MARKER: starting reads [ 4014.793838] Lustre: DEBUG MARKER: truncating the file [ 4015.233279] Lustre: DEBUG MARKER: killing dd [ 4015.646526] Lustre: DEBUG MARKER: removing the temporary file [ 4017.383440] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 18:13:20 (1747260800) [ 4019.129511] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 18:13:21 (1747260801) [ 4022.347676] Lustre: *** cfs_fail_loc=229, val=1*** [ 4022.349342] Lustre: Skipped 1 previous similar message [ 4023.067654] Lustre: *** cfs_fail_loc=229, val=1*** [ 4023.069528] Lustre: Skipped 1 previous similar message [ 4025.293179] Lustre: *** cfs_fail_loc=229, val=0*** [ 4031.909825] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 18:13:34 (1747260814) [ 4034.125315] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 18:13:36 (1747260816) [ 4036.082060] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 18:13:38 (1747260818) [ 4038.009982] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 18:13:40 (1747260820) [ 4039.922278] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 18:13:42 (1747260822) [ 4039.977790] Lustre: lustre-OST0001: Client 60078c12-876d-45c4-a072-9817ba1991a0 (at 192.168.201.23@tcp) reconnecting [ 4045.294961] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 18:13:48 (1747260828) [ 4046.518409] LustreError: 28345:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 sleeping for 3000ms [ 4049.521424] LustreError: 28345:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 awake [ 4051.945975] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 18:13:54 (1747260834) [ 4057.015789] Lustre: *** cfs_fail_loc=520, val=57344*** [ 4057.018118] LNet: *** cfs_fail_loc=e000, val=2147483648*** [ 4062.026363] Lustre: lustre-MDT0000: Client 60078c12-876d-45c4-a072-9817ba1991a0 (at 192.168.201.23@tcp) reconnecting [ 4062.035817] Lustre: 5872:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff88006696aac0 x1832134248239488/t34359738608(0) o35->60078c12-876d-45c4-a072-9817ba1991a0@192.168.201.23@tcp:0/0 lens 392/456 e 0 to 0 dl 1747260849 ref 1 fl Interpret:/2/0 rc 0/0 job:'' [ 4069.187755] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 18:14:11 (1747260851) [ 4070.455695] LustreError: 28345:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 sleeping for 22000ms [ 4090.071409] Lustre: lustre-OST0000: Client 60078c12-876d-45c4-a072-9817ba1991a0 (at 192.168.201.23@tcp) reconnecting [ 4091.358368] LustreError: 28345:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 4091.362300] LustreError: 28345:0:(ldlm_lib.c:3493:target_bulk_io()) @@@ bulk READ failed: rc = -107 req@ffff8801199b1c80 x1832134248245184/t0(0) o3->60078c12-876d-45c4-a072-9817ba1991a0@192.168.201.23@tcp:0/0 lens 488/440 e 0 to 0 dl 1747260872 ref 1 fl Interpret:/0/0 rc 0/0 job:'' [ 4091.367866] Lustre: 28345:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/1s); client may timeout req@ffff8801199b1c80 x1832134248245184/t0(0) o3->60078c12-876d-45c4-a072-9817ba1991a0@192.168.201.23@tcp:0/0 lens 488/440 e 0 to 0 dl 1747260872 ref 1 fl Complete:/0/0 rc 0/0 job:'' [ 4093.460280] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [ 4093.922995] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [ 4094.369282] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 18:14:37 (1747260877) [ 4096.225582] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 18:14:38 (1747260878) [ 4098.049130] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 18:14:40 (1747260880) [ 4098.161740] Lustre: lustre-MDT0000: subdir mount '/d226c.sanity' is remote and may be slow [ 4099.920553] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 18:14:42 (1747260882) [ 4101.704045] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 18:14:44 (1747260884) [ 4151.175128] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 18:15:33 (1747260933) [ 4185.516677] Lustre: Failing over lustre-MDT0000 [ 4185.518304] Lustre: Skipped 1 previous similar message [ 4185.579587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4185.582858] Lustre: Skipped 10 previous similar messages [ 4185.584005] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4185.585530] Lustre: Skipped 6 previous similar messages [ 4185.701582] LustreError: 4172:0:(ldlm_resource.c:1126:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200003ab1:0x1ecf:0x0].0x0 (ffff8800a0f6d700) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4185.799796] Lustre: server umount lustre-MDT0000 complete [ 4185.801014] Lustre: Skipped 1 previous similar message [ 4188.024285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4188.055705] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4188.055712] LustreError: 11-0: MGC192.168.201.123@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 4188.055721] LustreError: 166-1: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4188.056075] Lustre: Evicted from MGS (at 192.168.201.123@tcp) after server handle changed from 0xdc2b222c1bdbf488 to 0xdc2b222c1bf28ff2 [ 4188.060281] Lustre: MGC192.168.201.123@tcp: Connection restored to (at 0@lo) [ 4188.060283] Lustre: Skipped 9 previous similar messages [ 4188.078594] LustreError: Skipped 52 previous similar messages [ 4188.150990] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4188.154562] Lustre: Skipped 1 previous similar message [ 4188.302530] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4188.305576] Lustre: Skipped 1 previous similar message [ 4189.111118] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4193.153276] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4193.156277] Lustre: Skipped 1 previous similar message [ 4193.170104] Lustre: lustre-OST0000: deleting orphan objects from 0x0:26589 to 0x0:26625 [ 4193.170163] Lustre: lustre-OST0001: deleting orphan objects from 0x0:26128 to 0x0:26145 [ 4205.720718] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 18:16:28 (1747260988) [ 4278.090853] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 18:17:40 (1747261060) [ 4279.918353] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 18:17:42 (1747261062) [ 4281.757065] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 18:17:44 (1747261064) [ 4291.127093] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 18:17:53 (1747261073) [ 4291.561587] Lustre: *** cfs_fail_loc=1801, val=0*** [ 4291.563262] Lustre: Skipped 1 previous similar message [ 4291.822723] LustreError: 27493:0:(mdd_dir.c:4234:mdd_migrate_cmd_check()) lustre-MDD0000: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush migrate_dir' to finish migration. [ 4293.708920] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [ 4294.154121] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 18:17:56 (1747261076) [ 4296.042891] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 18:17:58 (1747261078) [ 4298.066555] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 18:18:00 (1747261080) [ 4299.715794] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 18:18:02 (1747261082) [ 4301.518933] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 18:18:04 (1747261084) [ 4303.304935] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 18:18:06 (1747261086) [ 4305.177530] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 18:18:07 (1747261087) [ 4305.575831] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [ 4306.067986] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 18:18:08 (1747261088) [ 4327.122509] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 18:18:29 (1747261109) [ 4329.940753] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 18:18:32 (1747261112) [ 4332.057001] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 18:18:34 (1747261114) [ 4338.190948] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 18:18:40 (1747261120) [ 4343.931384] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 18:18:46 (1747261126) [ 4353.759231] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 18:18:56 (1747261136) [ 4355.575639] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 18:18:58 (1747261138) [ 4358.440704] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 18:19:01 (1747261141) [ 4360.322346] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 18:19:03 (1747261143) [ 4360.692942] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [ 4361.134841] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 18:19:03 (1747261143) [ 4361.522884] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [ 4361.969043] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 18:19:04 (1747261144) [ 4363.762505] Lustre: DEBUG MARKER: == sanity test 230y: unlink dir with bad hash type ======= 18:19:06 (1747261146) [ 4365.163702] Lustre: *** cfs_fail_loc=1802, val=0*** [ 4365.670068] Lustre: *** cfs_fail_loc=1802, val=0*** [ 4365.671397] Lustre: Skipped 50 previous similar messages [ 4368.512290] Lustre: DEBUG MARKER: == sanity test 230z: resume dir migration with bad hash type ========================================================== 18:19:11 (1747261151) [ 4369.933977] Lustre: *** cfs_fail_loc=1802, val=0*** [ 4369.935491] Lustre: Skipped 18 previous similar messages [ 4376.902141] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 18:19:19 (1747261159) [ 4380.408995] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 18:19:23 (1747261163) [ 4385.120401] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 18:19:27 (1747261167) [ 4385.367745] Lustre: *** cfs_fail_loc=31c, val=0*** [ 4386.452360] Lustre: Failing over lustre-OST0000 [ 4386.480670] Lustre: server umount lustre-OST0000 complete [ 4388.235657] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4388.238104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.759923] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4388.763172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4388.806571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4388.809469] Lustre: Skipped 2 previous similar messages [ 4388.812481] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4389.578554] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4389.721941] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4389.926742] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4389.928824] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:32140 to 0x280000400:32161 [ 4389.929513] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28186 to 0x0:28225 [ 4391.955076] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 18:19:34 (1747261174) [ 4392.321760] Lustre: *** cfs_fail_loc=31c, val=0*** [ 4392.324202] LustreError: 28709:0:(ldlm_request.c:490:ldlm_cli_enqueue_local()) ### delayed lvb init failed (rc -12) ns: filter-lustre-OST0000_UUID lock: ffff88009e6be640/0xdc2b222c1c1135c5 lrc: 2/0,0 mode: --/PR res: [0x6e42:0x0:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 28709 timeout: 0 lvb_type: 0 [ 4395.955019] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4395.961188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4397.027807] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4397.291562] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28227 to 0x0:28257 [ 4397.292059] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:32140 to 0x280000400:32193 [ 4399.770265] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 18:19:42 (1747261182) [ 4402.196160] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 18:19:44 (1747261184) [ 4404.598161] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 18:19:47 (1747261187) [ 4406.872116] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 18:19:49 (1747261189) [ 4411.008884] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 18:19:53 (1747261193) [ 4413.007007] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 18:19:55 (1747261195) [ 4414.820165] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 18:19:57 (1747261197) [ 4438.922934] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 18:20:21 (1747261221) [ 4440.204953] Lustre: *** cfs_fail_loc=2100, val=0*** [ 4445.832266] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 18:20:28 (1747261228) [ 4452.125229] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 18:20:34 (1747261234) [ 4452.725702] Lustre: *** cfs_fail_loc=713, val=0*** [ 4452.727144] Lustre: Skipped 1 previous similar message [ 4457.495607] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 18:20:40 (1747261240) [ 4473.425422] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 18:20:56 (1747261256) [ 4479.292148] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 18:21:02 (1747261262) [ 4479.594350] Lustre: *** cfs_fail_loc=105, val=0*** [ 4481.452662] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 18:21:04 (1747261264) [ 4492.274174] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 18:21:14 (1747261274) [ 4502.187507] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 18:21:24 (1747261284) [ 4504.114671] Lustre: DEBUG MARKER: == sanity test 245: check mdc connection flag/data: multiple modify RPCs ========================================================== 18:21:26 (1747261286) [ 4506.354783] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 18:21:28 (1747261288) [ 4508.464613] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 18:21:31 (1747261291) [ 4510.216131] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 18:21:32 (1747261292) [ 4510.457436] Lustre: lustre-MDT0000: subdir mount '/d247c.sanity' is remote and may be slow [ 4512.975098] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 18:21:35 (1747261295) [ 4513.234041] Lustre: lustre-MDT0000: subdir mount '/d247d.sanity' is remote and may be slow [ 4515.067082] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 18:21:37 (1747261297) [ 4517.019437] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 18:21:39 (1747261299) [ 4518.509180] Lustre: lustre-MDT0000: subdir mount '/d247f.sanity/remote' refused because 'enable_remote_subdir_mount=0': rc = -66 [ 4521.070770] Lustre: lustre-MDT0000: subdir mount '/d247f.sanity/remote' is remote and may be slow [ 4523.469124] Lustre: DEBUG MARKER: == sanity test 247g: mount striped directory as fileset caches ROOT lookup lock ========================================================== 18:21:46 (1747261306) [ 4523.841819] Lustre: DEBUG MARKER: SKIP: sanity test_247g needs >= 4 MDTs [ 4524.275953] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 18:21:47 (1747261307) [ 4531.013380] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 18:21:53 (1747261313) [ 4539.083974] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 18:22:01 (1747261321) [ 4540.929963] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 18:22:03 (1747261323) [ 4542.858053] Lustre: DEBUG MARKER: == sanity test 251: Handling short read and write correctly ========================================================== 18:22:05 (1747261325) [ 4544.464407] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 18:22:07 (1747261327) [ 4548.041478] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 18:22:10 (1747261330) [ 4599.779251] LustreError: 27493:0:(lod_qos.c:1362:lod_ost_alloc_specific()) can't lstripe objid [0x200004283:0x2f:0x0]: have 0 want 1 [ 4642.469031] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 18:23:44 (1747261424) [ 4643.645989] Lustre: lustre-MDD0000: changelog on [ 4643.647081] Lustre: Skipped 1 previous similar message [ 4647.351987] Lustre: lustre-MDD0001: changelog off [ 4647.354098] Lustre: Skipped 1 previous similar message [ 4649.275123] Lustre: DEBUG MARKER: SKIP: sanity test_255a skipping excluded test 255a (base 255) [ 4649.982669] Lustre: DEBUG MARKER: SKIP: sanity test_255b skipping excluded test 255b (base 255) [ 4650.739312] Lustre: DEBUG MARKER: SKIP: sanity test_255c skipping excluded test 255c (base 255) [ 4651.445234] Lustre: DEBUG MARKER: SKIP: sanity test_256 skipping excluded test 256 [ 4652.136192] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 18:23:54 (1747261434) [ 4652.808893] Lustre: *** cfs_fail_loc=161, val=0*** [ 4653.454992] Lustre: Failing over lustre-MDT0001 [ 4653.457706] Lustre: Skipped 1 previous similar message [ 4653.852172] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4653.856913] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4653.863944] Lustre: Skipped 3 previous similar messages [ 4653.866820] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4653.870070] Lustre: Skipped 2 previous similar messages [ 4656.293547] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.23@tcp (stopping) [ 4658.695636] Lustre: server umount lustre-MDT0001 complete [ 4658.697419] Lustre: Skipped 1 previous similar message [ 4661.294617] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4661.304599] LustreError: Skipped 4 previous similar messages [ 4662.068741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4662.186852] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4662.194902] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 4662.197449] Lustre: Skipped 1 previous similar message [ 4663.440631] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4663.911249] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4663.915763] Lustre: Skipped 1 previous similar message [ 4665.336571] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 4667.197720] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4667.202257] Lustre: Skipped 8 previous similar messages [ 4667.212300] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4667.217105] Lustre: Skipped 1 previous similar message [ 4667.240834] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34695 to 0x280000400:34721 [ 4667.240846] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:34207 to 0x2c0000400:34241 [ 4673.067764] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 18:24:15 (1747261455) [ 4675.750902] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 18:24:18 (1747261458) [ 4677.926905] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 18:24:20 (1747261460) [ 4689.976936] Lustre: *** cfs_fail_loc=2301, val=0*** [ 4693.577186] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4693.581855] LDISKFS-fs (dm-2): 1 truncate cleaned up [ 4693.582872] LDISKFS-fs (dm-2): recovery complete [ 4693.585172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4693.626255] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4694.738994] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4695.453862] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28284 to 0x0:28321 [ 4695.454245] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34695 to 0x280000400:34753 [ 4700.026647] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 18:24:42 (1747261482) [ 4702.069704] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 18:24:44 (1747261484) [ 4706.491983] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 18:24:48 (1747261488) [ 4709.118011] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 18:24:51 (1747261491) [ 4711.490384] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 18:24:54 (1747261494) [ 4714.220887] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 18:24:56 (1747261496) [ 4717.458867] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 18:24:59 (1747261499) [ 4719.620873] Lustre: Increasing provided stripe size to a minimum value 64 [ 4724.516935] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 18:25:07 (1747261507) [ 4727.420140] Lustre: *** cfs_fail_loc=168, val=0*** [ 4728.156268] Lustre: *** cfs_fail_loc=168, val=0*** [ 4728.158749] Lustre: Skipped 2 previous similar messages [ 4729.327616] Lustre: *** cfs_fail_loc=168, val=0*** [ 4729.328821] Lustre: Skipped 5 previous similar messages [ 4734.329569] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 18:25:16 (1747261516) [ 4737.803265] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 18:25:20 (1747261520) [ 4740.461770] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 18:25:22 (1747261522) [ 4743.123690] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 18:25:25 (1747261525) [ 4745.852613] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 18:25:28 (1747261528) [ 4748.589411] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 18:25:31 (1747261531) [ 4779.874359] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 18:26:02 (1747261562) [ 4782.716273] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 18:26:05 (1747261565) [ 4785.837296] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 18:26:08 (1747261568) [ 4789.547937] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 18:26:12 (1747261572) [ 4792.472166] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 18:26:14 (1747261574) [ 4796.617428] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 18:26:19 (1747261579) [ 4800.340124] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 18:26:22 (1747261582) [ 4804.039215] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 18:26:26 (1747261586) [ 4806.766806] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 18:26:29 (1747261589) [ 4809.647830] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 18:26:32 (1747261592) [ 4811.671899] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 18:26:34 (1747261594) [ 4811.949016] LustreError: 7073:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 16b sleeping for 2000ms [ 4813.950437] LustreError: 7073:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 16b awake [ 4816.323809] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 18:26:38 (1747261598) [ 4820.475105] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 18:26:43 (1747261603) [ 4824.256650] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4824.264955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4824.325777] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4825.570975] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4825.959942] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:34817 [ 4825.960473] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28353 [ 4826.895071] LustreError: 20267:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4829.690747] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4829.696794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4829.767008] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4830.734960] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4831.475270] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:34849 [ 4831.480672] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28385 [ 4831.957794] LustreError: 22704:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4834.747043] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4834.751236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4834.804081] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4835.925251] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4836.462929] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28417 [ 4836.463687] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:34881 [ 4837.214563] LustreError: 25137:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4840.266400] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4840.272853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4841.286150] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4841.561765] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:34913 [ 4841.565161] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28449 [ 4842.330836] LustreError: 27598:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4844.970921] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4844.976890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4845.039677] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4845.043806] Lustre: Skipped 1 previous similar message [ 4846.069491] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4846.908543] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28481 [ 4846.909139] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:34945 [ 4847.140724] LustreError: 30127:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4847.144576] LustreError: 30127:0:(obd_class.h:1061:obd_statfs()) Skipped 1 previous similar message [ 4849.699220] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4849.703194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4850.601697] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4851.373305] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:34977 [ 4851.373800] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28513 [ 4854.722580] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4854.728155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4855.952505] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4856.662449] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35009 [ 4856.663081] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28545 [ 4857.172589] LustreError: 2743:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4857.176449] LustreError: 2743:0:(obd_class.h:1061:obd_statfs()) Skipped 1 previous similar message [ 4860.165916] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4860.172037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4861.194994] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4862.010817] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28577 [ 4862.011420] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35041 [ 4865.063823] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4865.068565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4865.116667] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4865.119540] Lustre: Skipped 3 previous similar messages [ 4866.009607] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4867.053006] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28609 [ 4867.056930] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35073 [ 4869.692764] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4869.697180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4870.708721] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4871.737460] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35105 [ 4871.740290] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28641 [ 4874.899185] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4874.907364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4875.990268] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4876.875064] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35137 [ 4876.875944] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28673 [ 4877.283084] LustreError: 12551:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4877.286237] LustreError: 12551:0:(obd_class.h:1061:obd_statfs()) Skipped 4 previous similar messages [ 4880.135221] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4880.141121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4881.358318] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4882.004788] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28705 [ 4882.008378] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35169 [ 4885.704284] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4885.710017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4886.691609] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4887.505452] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28737 [ 4887.505971] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35201 [ 4890.621265] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4890.629243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4891.956171] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4892.548889] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35233 [ 4892.553556] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28769 [ 4896.517319] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4896.524751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4897.678710] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4898.317314] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35265 [ 4898.318108] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28801 [ 4902.168537] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4902.175630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4902.235737] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4902.240195] Lustre: Skipped 6 previous similar messages [ 4903.459504] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4903.768427] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35297 [ 4903.768975] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28833 [ 4907.441803] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4907.445333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4908.356980] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4909.148151] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35329 [ 4909.152757] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28865 [ 4909.715047] LustreError: 27099:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 4909.718888] LustreError: 27099:0:(obd_class.h:1061:obd_statfs()) Skipped 6 previous similar messages [ 4912.790411] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4912.796510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4914.008527] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4914.345857] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35361 [ 4914.351054] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28897 [ 4918.418698] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4918.426409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4919.792599] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4920.035223] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28929 [ 4920.035392] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35393 [ 4924.177448] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4924.183559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4925.470489] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4925.800819] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28961 [ 4925.801555] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35425 [ 4929.161911] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 18:28:31 (1747261711) [ 4931.520829] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 18:28:34 (1747261714) [ 4932.394215] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -19 [ 4932.397983] LustreError: Skipped 1 previous similar message [ 4932.400865] LustreError: 2664:0:(osp_dev.c:494:osp_disconnect()) lustre-MDT0000-osp-MDT0001: can't disconnect: rc = -19 [ 4932.409073] LustreError: 2664:0:(lod_dev.c:261:lod_sub_process_config()) lustre-MDT0001-mdtlov: error cleaning up LOD index 0: cmd 0xcf031 : rc = -19 [ 4932.464634] LustreError: 2664:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 60c sleeping [ 4935.503311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4935.552717] LustreError: 3244:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 60c waking [ 4935.552956] LustreError: 166-1: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4935.553370] Lustre: Evicted from MGS (at 192.168.201.123@tcp) after server handle changed from 0xdc2b222c1bf28ff2 to 0xdc2b222c1c1c2a95 [ 4935.561015] LustreError: 2664:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 60c awake: rc=1906 [ 4937.654741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4937.658653] Lustre: Skipped 1 previous similar message [ 4938.932221] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4942.519496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4944.001163] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4945.780509] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 4947.700443] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:28993 [ 4947.700471] Lustre: lustre-OST0001: deleting orphan objects from 0x0:27736 to 0x0:27777 [ 4947.710881] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:34250 to 0x2c0000400:34273 [ 4947.710906] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34773 to 0x280000400:35457 [ 4953.369774] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 18:28:55 (1747261735) [ 4954.167454] LustreError: 3252:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 15e sleeping [ 4955.895610] LustreError: 5601:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4955.899667] LustreError: 3252:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 15e awake: rc=3271 [ 4956.907261] LustreError: 5601:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4960.410167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4960.466820] LustreError: 166-1: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4960.470819] Lustre: Evicted from MGS (at 192.168.201.123@tcp) after server handle changed from 0xdc2b222c1c1c2a95 to 0xdc2b222c1c1c3164 [ 4960.481162] LustreError: 6212:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4961.575684] LustreError: 6210:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4962.583560] LustreError: 6210:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4964.595404] LustreError: 6210:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4964.597820] LustreError: 6210:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message [ 4968.611081] LustreError: 6228:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4968.612826] LustreError: 6228:0:(libcfs_fail.h:180:cfs_race()) Skipped 3 previous similar messages [ 4971.987276] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4972.647451] Lustre: lustre-OST0001: deleting orphan objects from 0x0:27736 to 0x0:27809 [ 4972.647453] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28334 to 0x0:29025 [ 4976.855303] LustreError: 6228:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 4976.859592] LustreError: 6228:0:(libcfs_fail.h:180:cfs_race()) Skipped 5 previous similar messages [ 4980.731791] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 18:29:23 (1747261763) [ 4983.967047] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 18:29:26 (1747261766) [ 5007.297460] Lustre: DEBUG MARKER: == sanity test 300c: chown [ 5034.482356] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 18:30:17 (1747261817) [ 5037.212693] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ========================================================== 18:30:19 (1747261819) [ 5040.192000] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ========================================================== 18:30:22 (1747261822) [ 5043.117118] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ========================================================== 18:30:25 (1747261825) [ 5047.116439] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory ========================================================== 18:30:29 (1747261829) [ 5051.657286] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory ========================================================== 18:30:34 (1747261834) [ 5055.765809] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ============ 18:30:38 (1747261838) [ 5058.728897] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ======== 18:30:41 (1747261841) [ 5061.726934] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ========================================================== 18:30:44 (1747261844) [ 5063.913900] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS ========================================================== 18:30:46 (1747261846) [ 5064.512096] Lustre: DEBUG MARKER: SKIP: sanity test_300m Only for single MDT [ 5065.246186] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA ========================================================== 18:30:47 (1747261847) [ 5071.413979] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [ 5072.148633] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================================== 18:30:54 (1747261854) [ 5072.634924] Lustre: *** cfs_fail_loc=1704, val=0*** [ 5072.637237] LustreError: 27499:0:(out_handler.c:910:out_tx_end()) lustre-MDT0001-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524 [ 5075.250355] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ========================================================== 18:30:57 (1747261857) [ 5078.036881] Lustre: DEBUG MARKER: == sanity test 300r: test -1 striped directory =========== 18:31:00 (1747261860) [ 5078.088706] LustreError: 6252:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -2 [ 5078.096557] LustreError: 6252:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2 [ 5080.795573] Lustre: DEBUG MARKER: == sanity test 300s: test lfs mkdir -c without -i ======== 18:31:03 (1747261863) [ 5083.591054] Lustre: DEBUG MARKER: == sanity test 300t: test max_mdt_stripecount ============ 18:31:06 (1747261866) [ 5087.301111] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ============= 18:31:09 (1747261869) [ 5090.111062] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open ========================================================== 18:31:12 (1747261872) [ 5092.866952] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ========================================================== 18:31:15 (1747261875) [ 5093.465713] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [ 5094.225552] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ========================================================== 18:31:16 (1747261876) [ 5109.987590] Lustre: DEBUG MARKER: == sanity test 312: make sure ZFS adjusts its block size by write pattern ========================================================== 18:31:32 (1747261892) [ 5110.701618] Lustre: DEBUG MARKER: SKIP: sanity test_312 the test only applies to zfs [ 5111.467582] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ========================================================== 18:31:33 (1747261893) [ 5111.945287] Lustre: *** cfs_fail_loc=720, val=0*** [ 5111.947459] LustreError: 28346:0:(osd_handler.c:2093:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [ 5111.952541] LustreError: 28346:0:(ofd_objects.c:987:ofd_object_punch()) lustre-OST0000: failed to stop transaction: rc = -5 [ 5114.911538] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ========================================================== 18:31:37 (1747261897) [ 5116.135234] Lustre: *** cfs_fail_loc=720, val=0*** [ 5116.135361] LustreError: 28709:0:(osd_handler.c:2093:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [ 5116.135378] LustreError: 28709:0:(ofd_objects.c:1054:ofd_destroy()) lustre-OST0000 failed to stop transaction: -5 [ 5116.135496] LustreError: 28709:0:(ofd_dev.c:1837:ofd_destroy_hdl()) lustre-OST0000: error destroying object [0x100000000:0x7a7a:0x0]: -5 [ 5116.153921] Lustre: Skipped 1 previous similar message [ 5121.935240] Lustre: *** cfs_fail_loc=720, val=0*** [ 5121.937810] LustreError: 28346:0:(osd_handler.c:2093:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [ 5121.942892] LustreError: 28346:0:(osd_handler.c:2093:osd_trans_stop()) Skipped 1 previous similar message [ 5125.215171] Lustre: *** cfs_fail_loc=720, val=0*** [ 5125.217947] Lustre: Skipped 1 previous similar message [ 5125.220634] LustreError: 24107:0:(osd_handler.c:2093:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [ 5125.226513] LustreError: 24107:0:(osd_handler.c:2093:osd_trans_stop()) Skipped 1 previous similar message [ 5128.503535] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ============= 18:31:50 (1747261910) [ 5133.279535] Lustre: DEBUG MARKER: == sanity test 316: lfs migrate of file with large_xattr enabled ========================================================== 18:31:55 (1747261915) [ 5136.582521] Lustre: DEBUG MARKER: == sanity test 317: Verify blocks get correctly update after truncate ========================================================== 18:31:59 (1747261919) [ 5139.678932] Lustre: DEBUG MARKER: == sanity test 318: Verify async readahead tunables ====== 18:32:02 (1747261922) [ 5142.560450] Lustre: DEBUG MARKER: == sanity test 319: lost lease lock on migrate error ===== 18:32:05 (1747261925) [ 5150.438631] Lustre: DEBUG MARKER: == sanity test 350: force NID mismatch path to be exercised ========================================================== 18:32:12 (1747261932) [ 5151.060541] LNet: *** cfs_fail_loc=e001, val=0*** [ 5151.062859] LNet: 26055:0:(lib-ptl.c:184:lnet_try_match_md()) message from 12345-0@lo matched 1832134231136320 with NID mismatch 0@lo accepted (same peer ffff88012e3dc400) [ 5209.660704] Lustre: DEBUG MARKER: == sanity test 398a: direct IO should cancel lock otherwise lockless ========================================================== 18:33:12 (1747261992) [ 5212.521326] Lustre: DEBUG MARKER: == sanity test 398b: DIO and buffer IO race ============== 18:33:15 (1747261995) [ 5218.188979] Lustre: DEBUG MARKER: == sanity test 398c: run fio to test AIO ================= 18:33:20 (1747262000) [ 5232.136260] Lustre: DEBUG MARKER: == sanity test 398d: run aiocp to verify block size > stripe size ========================================================== 18:33:34 (1747262014) [ 5237.115182] Lustre: DEBUG MARKER: == sanity test 398e: O_Direct open cleared by fcntl doesn't cause hang ========================================================== 18:33:39 (1747262019) [ 5239.827453] Lustre: DEBUG MARKER: == sanity test 398f: verify aio handles ll_direct_rw_pages errors correctly ========================================================== 18:33:42 (1747262022) [ 5243.103270] Lustre: DEBUG MARKER: == sanity test 398g: verify parallel dio async RPC submission ========================================================== 18:33:45 (1747262025) [ 5243.903035] LustreError: 17219:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5243.908672] LustreError: 17219:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 7 previous similar messages [ 5245.903462] LustreError: 28345:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 5245.908426] LustreError: 28345:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 4 previous similar messages [ 5246.195611] LustreError: 24107:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5248.200436] LustreError: 24107:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 5248.205042] LustreError: 24107:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 5248.230088] LustreError: 24107:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5250.235458] LustreError: 24107:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 5252.374942] LustreError: 24107:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5252.379869] LustreError: 24107:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 5254.383444] LustreError: 24107:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 5254.387778] LustreError: 24107:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 5260.758358] LustreError: 24107:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 2000ms [ 5260.763200] LustreError: 24107:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 5262.767470] LustreError: 24107:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 5262.772750] LustreError: 24107:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 5265.888717] Lustre: DEBUG MARKER: == sanity test 398h: verify correctness of read [ 5271.019324] Lustre: DEBUG MARKER: == sanity test 398i: verify parallel dio handles ll_direct_rw_pages errors correctly ========================================================== 18:34:13 (1747262053) [ 5274.694656] Lustre: DEBUG MARKER: == sanity test 398j: test parallel dio where stripe size > rpc_size ========================================================== 18:34:17 (1747262057) [ 5279.754860] Lustre: DEBUG MARKER: == sanity test 398k: test enospc on first stripe ========= 18:34:22 (1747262062) [ 5293.304938] Lustre: DEBUG MARKER: SKIP: sanity test_398k 7020424 > 600000 skipping out-of-space test on OST0 [ 5294.080476] Lustre: DEBUG MARKER: == sanity test 398l: test enospc on intermediate stripe/RPC ========================================================== 18:34:36 (1747262076) [ 5299.887109] Lustre: DEBUG MARKER: SKIP: sanity test_398l 7000512 > 600000 skipping out-of-space test on OST0 [ 5300.548483] Lustre: DEBUG MARKER: == sanity test 398m: test RPC failures with parallel dio ========================================================== 18:34:43 (1747262083) [ 5301.050007] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5301.054377] Lustre: Skipped 7 previous similar messages [ 5302.075371] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5302.078107] Lustre: Skipped 3 previous similar messages [ 5304.502596] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5304.505080] Lustre: Skipped 3 previous similar messages [ 5307.576662] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5307.579137] Lustre: Skipped 3 previous similar messages [ 5311.614517] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5311.617250] Lustre: Skipped 3 previous similar messages [ 5322.644646] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5322.647295] Lustre: Skipped 7 previous similar messages [ 5346.667693] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5346.670736] Lustre: Skipped 11 previous similar messages [ 5379.622313] Lustre: *** cfs_fail_loc=20f, val=0*** [ 5379.623443] Lustre: Skipped 34 previous similar messages [ 5450.723797] Lustre: *** cfs_fail_loc=20e, val=0*** [ 5450.726556] Lustre: Skipped 52 previous similar messages [ 5530.113118] Lustre: DEBUG MARKER: == sanity test 398n: test append with parallel DIO ======= 18:38:32 (1747262312) [ 5534.531432] Lustre: DEBUG MARKER: == sanity test 399a: fake write should not be slower than normal write ========================================================== 18:38:37 (1747262317) [ 5552.776726] Lustre: *** cfs_fail_loc=238, val=0*** [ 5572.924048] Lustre: DEBUG MARKER: sanity test_399a: @@@@@@ IGNORE (env=kvm): fake write is slower