[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 461018771 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.592270] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.594397] pid_max: default: 32768 minimum: 301 [ 0.595792] Security Framework initialized [ 0.596900] SELinux: Initializing. [ 0.597788] SELinux: Starting in permissive mode [ 0.599428] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.607326] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.611411] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.615015] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.618102] Initializing cgroup subsys memory [ 0.619256] Initializing cgroup subsys devices [ 0.621266] Initializing cgroup subsys freezer [ 0.627845] Initializing cgroup subsys net_cls [ 0.629127] Initializing cgroup subsys blkio [ 0.631922] Initializing cgroup subsys perf_event [ 0.633034] Initializing cgroup subsys hugetlb [ 0.634055] Initializing cgroup subsys pids [ 0.635445] Initializing cgroup subsys net_prio [ 0.637682] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.640285] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.643154] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.644865] tlb_flushall_shift: 6 [ 0.646128] FEATURE SPEC_CTRL Present [ 0.647232] FEATURE IBPB_SUPPORT Present [ 0.648333] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.650488] Spectre V2 : Vulnerable [ 0.651508] Speculative Store Bypass: Vulnerable [ 0.654219] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.663118] ACPI: Core revision 20130517 [ 0.666357] ACPI: All ACPI Tables successfully acquired [ 0.668215] ftrace: allocating 30294 entries in 119 pages [ 0.773729] Enabling x2apic [ 0.774724] Enabled x2apic [ 0.781861] Switched APIC routing to physical x2apic. [ 0.786591] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.789448] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.796667] TSC deadline timer enabled [ 0.798295] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.805672] ... version: 2 [ 0.806967] ... bit width: 48 [ 0.815531] ... generic registers: 4 [ 0.816827] ... value mask: 0000ffffffffffff [ 0.818528] ... max period: 00007fffffffffff [ 0.827384] ... fixed-purpose events: 3 [ 0.828662] ... event mask: 000000070000000f [ 0.837556] KVM setup paravirtual spinlock [ 0.855137] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.864701] KVM setup async PF for cpu 1 [ 0.867492] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.875488] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.848291] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.879481] KVM setup async PF for cpu 2 [ 0.879944] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.889527] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.896280] KVM setup async PF for cpu 3 [ 0.897324] Brought up 4 CPUs [ 0.897326] smpboot: Max logical packages: 1 [ 0.897328] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.915862] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.922109] devtmpfs: initialized [ 0.923689] x86/mm: Memory block size: 128MB [ 0.930676] EVM: security.selinux [ 0.931941] EVM: security.ima [ 0.933059] EVM: security.capability [ 0.937949] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.941705] NET: Registered protocol family 16 [ 0.943748] cpuidle: using governor haltpoll [ 0.946291] ACPI: bus type PCI registered [ 0.947666] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.950287] PCI: Using configuration type 1 for base access [ 0.953454] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.975398] ACPI: Added _OSI(Module Device) [ 0.978125] ACPI: Added _OSI(Processor Device) [ 0.982060] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.986971] ACPI: Added _OSI(Processor Aggregator Device) [ 0.992712] ACPI: Added _OSI(Linux-Dell-Video) [ 0.996116] ACPI: EC: Look up EC in DSDT [ 1.005359] ACPI: Interpreter enabled [ 1.006761] ACPI: (supports S0 S3 S4 S5) [ 1.008009] ACPI: Using IOAPIC for interrupt routing [ 1.011921] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 1.019087] ACPI: Enabled 2 GPEs in block 00 to 0F [ 1.039956] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 1.045625] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 1.054172] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 1.063058] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 1.078978] acpiphp: Slot [2] registered [ 1.082780] acpiphp: Slot [3] registered [ 1.087089] acpiphp: Slot [4] registered [ 1.094396] acpiphp: Slot [5] registered [ 1.097795] acpiphp: Slot [6] registered [ 1.101590] acpiphp: Slot [7] registered [ 1.103046] acpiphp: Slot [8] registered [ 1.107876] acpiphp: Slot [9] registered [ 1.109400] acpiphp: Slot [10] registered [ 1.112500] acpiphp: Slot [11] registered [ 1.114162] acpiphp: Slot [12] registered [ 1.115609] acpiphp: Slot [13] registered [ 1.116874] acpiphp: Slot [14] registered [ 1.118439] acpiphp: Slot [15] registered [ 1.119802] acpiphp: Slot [16] registered [ 1.121223] acpiphp: Slot [17] registered [ 1.122661] acpiphp: Slot [18] registered [ 1.123984] acpiphp: Slot [19] registered [ 1.125283] acpiphp: Slot [20] registered [ 1.126514] acpiphp: Slot [21] registered [ 1.127914] acpiphp: Slot [22] registered [ 1.129244] acpiphp: Slot [23] registered [ 1.130666] acpiphp: Slot [24] registered [ 1.131945] acpiphp: Slot [25] registered [ 1.133328] acpiphp: Slot [26] registered [ 1.134826] acpiphp: Slot [27] registered [ 1.136314] acpiphp: Slot [28] registered [ 1.137745] acpiphp: Slot [29] registered [ 1.139149] acpiphp: Slot [30] registered [ 1.140459] acpiphp: Slot [31] registered [ 1.141874] PCI host bridge to bus 0000:00 [ 1.143379] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 1.145873] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 1.148081] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 1.150663] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 1.153097] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 1.155559] pci_bus 0000:00: root bus resource [bus 00-ff] [ 1.157482] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 1.158323] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 1.159393] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 1.167610] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 1.170797] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 1.173934] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 1.176203] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 1.181035] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 1.184469] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 1.185120] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 1.197081] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 1.204859] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 1.209043] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 1.228942] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 1.230980] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 1.239506] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 1.249652] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 1.259839] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 1.284903] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 1.294725] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.297757] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.300438] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.311286] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 1.329519] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.341538] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.352724] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.388462] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 1.403385] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.411318] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.419979] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.439302] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 1.450389] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.467281] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.479877] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.524952] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 1.536916] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.539927] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.545145] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.563765] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 1.575895] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.585862] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.591075] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.611803] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.616194] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.626403] vgaarb: loaded [ 1.628151] SCSI subsystem initialized [ 1.629381] ACPI: bus type USB registered [ 1.635768] usbcore: registered new interface driver usbfs [ 1.637312] usbcore: registered new interface driver hub [ 1.638813] usbcore: registered new device driver usb [ 1.651008] PCI: Using ACPI for IRQ routing [ 1.655703] PCI: pci_cache_line_size set to 64 bytes [ 1.656021] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.656028] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.656030] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.660767] NetLabel: Initializing [ 1.664972] NetLabel: domain hash size = 128 [ 1.669554] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.674629] NetLabel: unlabeled traffic allowed by default [ 1.678745] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.683662] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.697237] amd_nb: Cannot enumerate AMD northbridges [ 1.710707] Switched to clocksource kvm-clock [ 1.776532] pnp: PnP ACPI init [ 1.781732] ACPI: bus type PNP registered [ 1.782934] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.783028] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.783079] pnp 00:02: [dma 2] [ 1.783113] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.783202] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.783273] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.783345] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.785250] pnp: PnP ACPI: found 6 devices [ 1.792679] ACPI: bus type PNP unregistered [ 1.837706] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.837713] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.837716] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.837719] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.837722] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.837800] NET: Registered protocol family 2 [ 1.855499] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.860574] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.874570] TCP: Hash tables configured (established 32768 bind 32768) [ 1.882596] TCP: reno registered [ 1.883580] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.885406] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.902274] NET: Registered protocol family 1 [ 1.917132] RPC: Registered named UNIX socket transport module. [ 1.925226] RPC: Registered udp transport module. [ 1.926449] RPC: Registered tcp transport module. [ 1.934684] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.946514] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.948552] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.962983] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.970375] PCI: CLS 0 bytes, default 64 [ 1.979191] Unpacking initramfs... [ 5.326737] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 5.334664] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.338515] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 5.349196] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 5.353658] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 5.356606] RAPL PMU: hw unit of domain package 2^-0 Joules [ 5.359021] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 5.370268] cryptomgr_test (52) used greatest stack depth: 14480 bytes left [ 5.374136] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 5.374194] Initialise system trusted keyring [ 5.453895] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 5.460206] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 5.480376] zpool: loaded [ 5.481829] zbud: loaded [ 5.483009] VFS: Disk quotas dquot_6.6.0 [ 5.487215] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 5.495477] NFS: Registering the id_resolver key type [ 5.497139] Key type id_resolver registered [ 5.501680] Key type id_legacy registered [ 5.502821] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 5.512327] Key type big_key registered [ 5.514037] SELinux: Registering netfilter hooks [ 5.528919] cryptomgr_test (58) used greatest stack depth: 14048 bytes left [ 5.542331] cryptomgr_test (63) used greatest stack depth: 13984 bytes left [ 5.554946] NET: Registered protocol family 38 [ 5.554960] Key type asymmetric registered [ 5.554964] Asymmetric key parser 'x509' registered [ 5.555118] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 5.568196] io scheduler noop registered [ 5.568202] io scheduler deadline registered (default) [ 5.568327] io scheduler cfq registered [ 5.568333] io scheduler mq-deadline registered [ 5.568339] io scheduler kyber registered [ 5.578090] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 5.578119] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 5.638710] intel_idle: does not run on family 6 model 62 [ 5.638896] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.645952] ACPI: Power Button [PWRF] [ 5.650037] GHES: HEST is not enabled! [ 5.795972] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 5.921164] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 6.147893] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 6.320929] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 6.365243] tsc: Refined TSC clocksource calibration: 2399.986 MHz [ 6.581710] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.670601] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 6.764486] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 6.780342] Non-volatile memory driver v1.3 [ 6.784169] Linux agpgart interface v0.103 [ 6.789704] crash memory driver: version 1.1 [ 6.791891] nbd: registered device at major 43 [ 6.812435] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 6.812497] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 6.812537] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 6.812573] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 6.812619] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 6.828767] virtio_blk virtio1: [vda] 115368 512-byte logical blocks (59.0 MB/56.3 MiB) [ 6.842611] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 6.842657] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 6.842693] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 6.842728] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 6.842773] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 6.859142] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.874397] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 6.874455] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 6.874536] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 6.874578] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 6.874617] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 6.903648] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.914543] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 6.914600] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 6.914642] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 6.914683] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 6.914727] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 6.924814] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.936238] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 6.936283] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 6.936326] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 6.936364] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 6.936404] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 6.952569] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.967704] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 6.967767] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 6.967811] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 6.967851] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 6.967889] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 6.978048] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.986552] rdac: device handler registered [ 6.995660] hp_sw: device handler registered [ 6.996992] emc: device handler registered [ 6.999798] libphy: Fixed MDIO Bus: probed [ 7.005985] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 7.010217] ehci-pci: EHCI PCI platform driver [ 7.011420] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 7.015406] ohci-pci: OHCI PCI platform driver [ 7.016917] uhci_hcd: USB Universal Host Controller Interface driver [ 7.023137] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.029853] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.031589] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.037342] mousedev: PS/2 mouse device common for all mice [ 7.045336] rtc_cmos 00:05: RTC can wake from S4 [ 7.048230] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.049739] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 7.051776] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.074255] hidraw: raw HID events driver (C) Jiri Kosina [ 7.076067] usbcore: registered new interface driver usbhid [ 7.085791] usbhid: USB HID core driver [ 7.094561] drop_monitor: Initializing network drop monitor service [ 7.096436] Netfilter messages via NETLINK v0.30. [ 7.108145] TCP: cubic registered [ 7.112395] Initializing XFRM netlink socket [ 7.114262] NET: Registered protocol family 10 [ 7.119833] NET: Registered protocol family 17 [ 7.121442] Key type dns_resolver registered [ 7.127271] mce: Using 10 MCE banks [ 7.139784] Loading compiled-in X.509 certificates [ 7.142504] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 7.158004] registered taskstats version 1 [ 7.169244] modprobe (72) used greatest stack depth: 13440 bytes left [ 7.203737] Key type trusted registered [ 7.223154] Key type encrypted registered [ 7.232805] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 7.244941] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 7.261385] rtc_cmos 00:05: setting system clock to 2025-02-14 03:28:47 UTC (1739503727) [ 7.280438] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 7.290938] Write protecting the kernel read-only data: 12288k [ 7.299343] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 7.318227] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 7.360027] random: systemd: uninitialized urandom read (16 bytes read) [ 7.372393] random: systemd: uninitialized urandom read (16 bytes read) [ 7.379069] random: systemd: uninitialized urandom read (16 bytes read) [ 7.392348] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 7.405563] systemd[1]: Detected virtualization kvm. [ 7.409822] systemd[1]: Detected architecture x86-64. [ 7.411372] systemd[1]: Running in initial RAM disk. [ 7.431084] systemd[1]: No hostname configured. [ 7.434716] systemd[1]: Set hostname to . [ 7.442284] random: systemd: uninitialized urandom read (16 bytes read) [ 7.449854] systemd[1]: Initializing machine ID from random generator. [ 7.711665] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 7.728875] random: systemd: uninitialized urandom read (16 bytes read) [ 7.731008] random: systemd: uninitialized urandom read (16 bytes read) [ 7.734609] random: systemd: uninitialized urandom read (16 bytes read) [ 7.737183] random: systemd: uninitialized urandom read (16 bytes read) [ 7.742809] random: systemd: uninitialized urandom read (16 bytes read) [ 7.744653] random: systemd: uninitialized urandom read (16 bytes read) [ 7.786551] systemd[1]: Created slice Root Slice. [ 7.794414] systemd[1]: Created slice System Slice. [ 7.802462] systemd[1]: Listening on Journal Socket. [ 7.810668] systemd[1]: Starting Load Kernel Modules... [ 7.826860] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 7.842070] systemd[1]: Starting dracut cmdline hook... [ 7.850994] systemd[1]: Reached target Timers. [ 7.862210] systemd[1]: Starting Setup Virtual Console... [ 7.869694] systemd[1]: Reached target Swap. [ 7.884407] systemd[1]: Listening on udev Kernel Socket. [ 7.903300] systemd[1]: Reached target Slices. [ 7.916812] systemd[1]: Reached target Local File Systems. [ 7.927357] systemd[1]: Listening on udev Control Socket. [ 7.945495] systemd[1]: Reached target Sockets. [ 7.969693] systemd[1]: Starting Journal Service... [ 7.999220] systemd[1]: Started Load Kernel Modules. [ 8.044821] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 8.060731] systemd[1]: Started Setup Virtual Console. [ 8.089027] systemd[1]: Starting Create Static Device Nodes in /dev... [ 8.110582] systemd[1]: Starting Apply Kernel Variables... [ 8.126933] systemd[1]: Started Journal Service. [ 8.187143] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 9.713137] random: fast init done [ 9.876672] libata version 3.00 loaded. [ 10.298154] ata_piix 0000:00:01.1: version 2.13 [ 10.354981] scsi host0: ata_piix [ 10.359407] scsi host1: ata_piix [ 10.365002] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.374684] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.648660] ip (321) used greatest stack depth: 13080 bytes left [ 10.780629] ip (344) used greatest stack depth: 12336 bytes left [ 15.059785] hrtimer: interrupt took 4314831 ns [ 15.360868] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 16.629769] systemd-journald[119]: Received SIGTERM from PID 1 (systemd). [ 17.343219] SELinux: Disabled at runtime. [ 17.346188] SELinux: Unregistering netfilter hooks [ 17.475426] ip_tables: (C) 2000-2006 Netfilter Core Team [ 17.482619] systemd[1]: Inserted module 'ip_tables' [ 19.900000] systemd-journald[570]: Received request to flush runtime journal from PID 1 [ 20.674958] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 21.016620] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.237966] cryptd: max_cpu_qlen set to 1000 [ 21.861591] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 21.988046] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 22.231456] AVX version of gcm_enc/dec engaged. [ 22.233108] AES CTR mode by8 optimization enabled [ 22.352966] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 22.383319] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 23.010122] EDAC MC: Ver: 3.0.0 [ 23.093511] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 23.093524] EDAC sbridge: Ver: 1.1.2 [ 44.059126] mount.nfs (780) used greatest stack depth: 10704 bytes left [ 61.908322] device-mapper: uevent: version 1.0.3 [ 61.913447] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 70.209440] libcfs: loading out-of-tree module taints kernel. [ 70.213067] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 70.285014] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 78.053234] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 78.802134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 78.811820] alg: No test for adler32 (adler32-zlib) [ 79.874586] Lustre: Lustre: Build Version: 2.16.52_26_g7793934 [ 80.155350] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 80.159278] LNet: Accept secure, port 988 [ 81.764759] Key type lgssc registered [ 82.445273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 88.443758] icp: module license 'CDDL' taints kernel. [ 88.445382] Disabling lock debugging due to kernel taint [ 90.846217] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 95.486580] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.621619] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.935865] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 107.948716] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.923606] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 111.932606] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.575238] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 120.386443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 120.420632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 120.435682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 121.553485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 121.571572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 121.611041] Lustre: lustre-MDT0000: new disk, initializing [ 121.648745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 121.657100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 121.707193] mount.lustre (7248) used greatest stack depth: 10208 bytes left [ 122.771247] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 127.725950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 127.755001] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 127.788032] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 127.796225] Lustre: 8404:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 127.807307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 127.810123] Lustre: Skipped 1 previous similar message [ 127.848690] Lustre: lustre-MDT0001: new disk, initializing [ 127.871911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 127.886795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 127.890154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 129.121079] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 131.802002] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 136.898167] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 136.922787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.979134] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 136.992838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 137.100079] Lustre: 9604:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 137.119026] Lustre: lustre-OST0000: new disk, initializing [ 137.123299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 137.158233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.491958] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 143.223071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 143.227240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 143.305550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 147.324923] random: crng init done [ 147.350069] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 147.375598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 147.445361] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 147.456389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 147.513343] Lustre: 10878:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 147.532587] Lustre: lustre-OST0001: new disk, initializing [ 147.534745] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 147.587507] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 150.648706] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 152.889571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 152.898663] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 152.952972] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 158.175251] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 162.913680] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 170.886139] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing check_logdir /tmp/testlogs/ [ 172.889248] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing yml_node [ 175.740418] Lustre: DEBUG MARKER: Client: 2.16.52.26 [ 177.206521] Lustre: DEBUG MARKER: MDS: 2.16.52.26 [ 178.438714] Lustre: DEBUG MARKER: OSS: 2.16.52.26 [ 179.256790] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Thu Feb 13 22:31:39 EST 2025 [ 189.494553] Lustre: DEBUG MARKER: - need mds1 <= 2.14.55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 190.335894] Lustre: DEBUG MARKER: - need mds1 < v2_14_55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 191.288074] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42c 42b 118c 118d 407 119i 851 411b [ 192.074257] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 [ 193.035557] Lustre: DEBUG MARKER: === sanity: start setup 22:31:53 (1739503913) === [ 195.214266] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing check_config_client /mnt/lustre [ 205.456576] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 207.303434] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 208.675218] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.401181] Lustre: DEBUG MARKER: === sanity: finish setup 22:32:11 (1739503931) === [ 214.166063] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 22:32:14 (1739503934) [ 216.072454] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 216.945334] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 22:32:17 (1739503937) [ 220.447922] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 22:32:21 (1739503941) [ 339.324213] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 22:34:19 (1739504059) [ 343.137922] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 22:34:23 (1739504063) [ 343.814890] Lustre: *** cfs_fail_loc=15b, val=0*** [ 343.817050] Lustre: *** cfs_fail_loc=15b, val=0*** [ 347.655179] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 22:34:28 (1739504068) [ 351.475423] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 22:34:32 (1739504072) [ 352.098428] Lustre: *** cfs_fail_loc=19a, val=0*** [ 352.605246] Lustre: *** cfs_fail_loc=19a, val=0*** [ 353.715424] Lustre: *** cfs_fail_loc=19a, val=0*** [ 353.718191] Lustre: Skipped 1 previous similar message [ 356.124641] Lustre: *** cfs_fail_loc=19a, val=0*** [ 356.126383] Lustre: Skipped 4 previous similar messages [ 360.570869] Lustre: *** cfs_fail_loc=19a, val=0*** [ 360.573782] Lustre: Skipped 8 previous similar messages [ 368.744418] Lustre: *** cfs_fail_loc=19a, val=0*** [ 368.747342] Lustre: Skipped 14 previous similar messages [ 385.204976] Lustre: *** cfs_fail_loc=19a, val=0*** [ 385.207010] Lustre: Skipped 28 previous similar messages [ 415.380732] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 22:35:35 (1739504135) [ 415.871896] Lustre: *** cfs_fail_loc=188, val=0*** [ 417.627999] Lustre: *** cfs_fail_loc=189, val=0*** [ 422.544148] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 423.584980] Lustre: DEBUG MARKER: == sanity test 60j: llog_reader reports corruptions ====== 22:35:44 (1739504144) [ 426.902831] Lustre: lustre-MDD0000: changelog on [ 428.429507] Lustre: lustre-MDD0001: changelog on [ 433.259620] Lustre: DEBUG MARKER: SKIP: sanity test_60j path oi.1/0x1:0xb:0x0 is not in 'O/1/d/' format [ 434.676992] Lustre: lustre-MDD0001: changelog off [ 436.018934] Lustre: lustre-MDD0000: changelog off [ 437.336704] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 22:35:57 (1739504157) [ 440.783971] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 22:36:01 (1739504161) [ 443.988197] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 22:36:04 (1739504164) [ 508.420428] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 22:37:09 (1739504229) [ 515.958247] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 22:37:16 (1739504236) [ 520.054719] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 521.035679] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 22:37:21 (1739504241) [ 524.940626] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 22:37:25 (1739504245) [ 551.331463] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 22:37:51 (1739504271) [ 552.752986] Lustre: *** cfs_fail_loc=725, val=0*** [ 556.438163] Lustre: *** cfs_fail_loc=725, val=0*** [ 560.828965] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 22:38:01 (1739504281) [ 567.169102] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 22:38:07 (1739504287) [ 572.882373] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 22:38:13 (1739504293) [ 585.815442] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 22:38:26 (1739504306) [ 590.004843] Lustre: *** cfs_fail_loc=513, val=17*** [ 590.006454] LustreError: 17168:0:(service.c:2122:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1824001935463424 [ 590.952583] Lustre: Failing over lustre-OST0000 [ 591.058053] Lustre: server umount lustre-OST0000 complete [ 593.649941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 598.658018] LustreError: 17171:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 598.665670] LustreError: 17171:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 6 previous similar messages [ 603.250649] LustreError: 17171:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.623544] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 604.632088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 604.676039] Lustre: 32643:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 604.770594] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 604.780098] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 605.797445] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 606.706117] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 606.706780] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.156@tcp (at 0@lo) [ 606.916623] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [ 610.525723] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 611.237039] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 616.000308] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 22:38:56 (1739504336) [ 618.299877] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 22:38:59 (1739504339) [ 620.479513] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 22:39:01 (1739504341) [ 623.300752] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 22:39:03 (1739504343) [ 627.007559] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 22:39:07 (1739504347) [ 630.027001] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 22:39:10 (1739504350) [ 632.426783] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 22:39:13 (1739504353) [ 635.397048] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 22:39:16 (1739504356) [ 638.206723] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 22:39:18 (1739504358) [ 641.132042] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 22:39:22 (1739504362) [ 644.310668] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 22:39:25 (1739504365) [ 645.010731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 645.018372] Lustre: Skipped 1 previous similar message [ 645.021858] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 645.025810] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.156@tcp (at 0@lo) [ 645.029455] Lustre: Skipped 1 previous similar message [ 645.700254] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 645.703279] Lustre: Skipped 1 previous similar message [ 652.536592] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 658.770275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.774304] Lustre: Skipped 3 previous similar messages [ 658.777061] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 658.780634] Lustre: Skipped 1 previous similar message [ 658.784668] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 658.788337] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.156@tcp (at 0@lo) [ 658.792116] Lustre: Skipped 3 previous similar messages [ 660.171970] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 660.239816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 661.416455] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 661.519675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 669.121782] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 675.967198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.974528] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 675.978382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 675.983642] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.156@tcp (at 0@lo) [ 677.152454] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 677.240083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 678.578958] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 678.729270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 685.766745] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 691.558398] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 691.563779] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 691.567174] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 691.572605] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.203.156@tcp (at 0@lo) [ 692.646869] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 692.721674] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 693.897540] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 693.976724] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 700.424810] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 707.127227] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.135489] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 707.140544] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 707.147995] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.203.156@tcp (at 0@lo) [ 708.332562] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 708.403135] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 709.666048] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 709.748831] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 711.975296] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 22:40:32 (1739504432) [ 714.569167] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 22:40:35 (1739504435) [ 716.674277] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 22:40:37 (1739504437) [ 731.842102] Lustre: DEBUG MARKER: == sanity test 65o: pool inheritance for mdt component === 22:40:52 (1739504452) [ 749.872446] Lustre: DEBUG MARKER: == sanity test 65p: setstripe with yaml file and huge number ========================================================== 22:41:10 (1739504470) [ 752.530566] Lustre: DEBUG MARKER: == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 22:41:13 (1739504473) [ 754.639620] Lustre: DEBUG MARKER: == sanity test 65r: prevent all-zero offsets ============= 22:41:15 (1739504475) [ 756.919521] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 22:41:17 (1739504477) [ 760.208236] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 22:41:21 (1739504481) [ 760.820683] Lustre: *** cfs_fail_loc=217, val=0*** [ 760.822716] Lustre: Skipped 1 previous similar message [ 761.842308] Lustre: *** cfs_fail_loc=217, val=0*** [ 763.956491] Lustre: DEBUG MARKER: == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 22:41:24 (1739504484) [ 768.258049] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 768.815328] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 22:41:29 (1739504489) [ 771.420069] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 22:41:32 (1739504492) [ 774.569412] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 22:41:35 (1739504495) [ 802.791797] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 22:42:03 (1739504523) [ 804.807720] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 22:42:05 (1739504525) [ 806.679892] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 22:42:07 (1739504527) [ 808.514672] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 22:42:09 (1739504529) [ 835.236876] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 22:42:36 (1739504556) [ 855.965906] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 22:42:56 (1739504576) [ 858.587775] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 22:42:59 (1739504579) [ 858.839358] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.56@tcp inode [0x200000407:0xc1d:0x0] object 0x280000401:3838 extent [0-4194303]: client csum 37ac01b5, server csum 37ac01b4 [ 860.469265] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 861.633414] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.56@tcp inode [0x200000407:0xc1d:0x0] object 0x280000401:3838 extent [0-4194303], client returned csum 73d6c879 (type 1), server csum 832537b8 (type 1) [ 862.454414] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 863.603928] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.56@tcp inode [0x200000407:0xc1d:0x0] object 0x280000401:3838 extent [0-4194303], client returned csum 1330b44c (type 2), server csum ea34b3fd (type 2) [ 864.443518] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 865.592626] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.56@tcp inode [0x200000407:0xc1d:0x0] object 0x280000401:3838 extent [0-4194303], client returned csum dc819c1c (type 4), server csum 37ac01b4 (type 4) [ 866.472937] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 867.617949] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.56@tcp inode [0x200000407:0xc1d:0x0] object 0x280000401:3838 extent [0-4194303], client returned csum 23e20ac6 (type 10), server csum 217f0a16 (type 10) [ 868.434420] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 870.354117] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 872.238619] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 873.375382] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.56@tcp inode [0x200000407:0xc1d:0x0] object 0x280000401:3838 extent [0-4194303], client returned csum 523d1385 (type 80), server csum c1e212b3 (type 80) [ 873.382027] LustreError: Skipped 2 previous similar messages [ 874.022652] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 875.642351] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 22:43:16 (1739504596) [ 877.971678] Lustre: 9616:0:(tgt_handler.c:1931:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000407:0xc1e:0x0]:[0-1048575]-bed9b96-8f042f1f [ 877.979245] LustreError: dumping log to /tmp/lustre-log.1739504598.9616 [ 887.308296] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 22:43:28 (1739504608) [ 887.439683] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.56@tcp inode [0x200000407:0xc20:0x0] object 0x2c0000401:3829 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 889.680214] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.56@tcp inode [0x200000407:0xc20:0x0] object 0x2c0000401:3829 extent [0-4194303], client returned csum f5a99216 (type 4), server csum de2bf73e (type 4) [ 889.689602] LustreError: Skipped 1 previous similar message [ 891.607211] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 22:43:32 (1739504612) [ 892.092043] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 892.193715] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.56@tcp inode [0x200000407:0xc21:0x0] object 0x280000401:3840 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 895.326365] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.56@tcp inode [0x200000407:0xc21:0x0] object 0x280000401:3840 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 895.333720] LustreError: Skipped 4 previous similar messages [ 902.462462] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.56@tcp inode [0x200000407:0xc21:0x0] object 0x280000401:3840 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 902.471924] LustreError: Skipped 2 previous similar messages [ 913.574627] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.56@tcp inode [0x200000407:0xc21:0x0] object 0x280000401:3840 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 913.581999] LustreError: Skipped 3 previous similar messages [ 937.747119] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.56@tcp inode [0x200000407:0xc21:0x0] object 0x280000401:3840 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 937.753816] LustreError: Skipped 5 previous similar messages [ 948.266487] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 976.741045] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.56@tcp inode [0x200000407:0xc21:0x0] object 0x280000401:3840 extent [0-4194303]: client csum 7d33b9a0, server csum 7d33b99f [ 976.749625] LustreError: Skipped 17 previous similar messages [ 1004.370400] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1049.184178] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.56@tcp inode [0x200000407:0xc21:0x0] object 0x280000401:3840 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 1049.191246] LustreError: Skipped 24 previous similar messages [ 1059.684024] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1116.759623] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1173.481814] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1179.997447] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.56@tcp inode [0x200000407:0xc21:0x0] object 0x280000401:3840 extent [4194304-8388607]: client csum a30822f0, server csum a30822ef [ 1180.012291] LustreError: Skipped 54 previous similar messages [ 1230.885725] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1288.284433] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1293.035753] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 22:50:13 (1739505013) [ 1294.145602] Lustre: *** cfs_fail_loc=21a, val=0*** [ 1297.524917] Lustre: *** cfs_fail_loc=21b, val=0*** [ 1305.449260] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 22:50:25 (1739505025) [ 1306.077179] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1307.939418] Lustre: Modifying parameter lustre.osc.lustre*.checksums in log params [ 1310.949121] Lustre: Disabling parameter lustre.osc.lustre*.checksums in log params [ 1318.174141] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1321.425554] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 22:50:41 (1739505041) [ 1322.597128] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 1323.853677] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1326.517655] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid 50 [ 1327.420584] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid in IDLE state after 0 sec [ 1329.963993] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid 50 [ 1331.000619] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid in FULL state after 0 sec [ 1332.139074] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1334.446939] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid 50 [ 1345.156774] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid in IDLE state after 9 sec [ 1347.524009] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid 50 [ 1348.500701] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid in FULL state after 0 sec [ 1349.686265] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1352.038696] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid 50 [ 1366.082437] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid in IDLE state after 12 sec [ 1368.564227] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid 50 [ 1369.488324] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid in FULL state after 0 sec [ 1370.726405] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1373.561250] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid 50 [ 1385.035687] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid in IDLE state after 10 sec [ 1387.322283] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid 50 [ 1388.348914] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid in FULL state after 0 sec [ 1389.491606] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1392.099079] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid 50 [ 1400.454994] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid in IDLE state after 7 sec [ 1402.664040] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid 50 [ 1403.859044] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid in FULL state after 0 sec [ 1404.886125] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1407.975902] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid 50 [ 1420.502635] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid in IDLE state after 11 sec [ 1422.791552] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid 50 [ 1423.672890] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid in FULL state after 0 sec [ 1424.803348] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1426.993101] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid 50 [ 1440.478574] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid in IDLE state after 12 sec [ 1442.471443] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid 50 [ 1443.229225] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b7c37800.ost_server_uuid in FULL state after 0 sec [ 1447.181282] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1448.227662] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 22:52:48 (1739505168) [ 1452.008041] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 22:52:52 (1739505172) [ 1453.264080] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1454.206543] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1455.161724] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1456.134310] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1460.162095] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1461.272878] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 22:53:01 (1739505181) [ 1469.369509] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 22:53:09 (1739505189) [ 1475.017317] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ================================================================================= 22:53:15 (1739505195) [ 1489.218739] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 22:53:29 (1739505209) [ 1494.204854] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 22:53:34 (1739505214) [ 1494.931596] Lustre: *** cfs_fail_loc=228, val=0*** [ 1498.504670] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 22:53:39 (1739505219) [ 1499.218068] Lustre: *** cfs_fail_loc=228, val=0*** [ 1502.811223] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 22:53:43 (1739505223) [ 1518.279674] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ========================================================== 22:53:58 (1739505238) [ 1521.469902] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 22:54:02 (1739505242) [ 1618.396351] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 22:55:39 (1739505339) [ 1630.148927] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 22:55:50 (1739505350) [ 1664.522321] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 22:56:25 (1739505385) [ 1860.734040] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 22:59:41 (1739505581) [ 1905.638569] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 23:00:26 (1739505626) [ 1910.698076] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 23:00:31 (1739505631) [ 1965.215153] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 23:01:25 (1739505685) [ 1976.336339] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 23:01:36 (1739505696) [ 1982.654115] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 23:01:43 (1739505703) [ 2019.602095] Lustre: DEBUG MARKER: == sanity test 101m: read ahead for small file and last stripe of the file ========================================================== 23:02:20 (1739505740) [ 2027.148074] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 23:02:27 (1739505747) [ 2031.798913] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 23:02:32 (1739505752) [ 2039.570241] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 23:02:39 (1739505759) [ 2043.541101] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 23:02:44 (1739505764) [ 2067.044508] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 23:03:07 (1739505787) [ 2088.980716] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 23:03:29 (1739505809) [ 2089.927273] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 2090.875150] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 2091.847052] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 2092.729761] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2096.394969] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 23:03:36 (1739505816) [ 2098.022077] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2098.994338] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 2100.112675] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 2101.300854] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2102.423656] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2106.285521] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 23:03:46 (1739505826) [ 2110.403354] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 23:03:50 (1739505830) [ 2131.734299] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 23:04:12 (1739505852) [ 2135.811885] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 23:04:16 (1739505856) [ 2139.562174] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 23:04:20 (1739505860) [ 2142.605688] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 23:04:23 (1739505863) [ 2146.412665] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 23:04:27 (1739505867) [ 2149.904711] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 23:04:30 (1739505870) [ 2153.930805] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 23:04:34 (1739505874) [ 2158.341743] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 23:04:38 (1739505878) [ 2162.673166] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 23:04:43 (1739505883) [ 2167.251409] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 23:04:47 (1739505887) [ 2289.748152] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 23:06:50 (1739506010) [ 2369.069780] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 23:08:09 (1739506089) [ 2373.327609] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 23:08:13 (1739506093) [ 2561.235915] Lustre: lustre-MDT0000: Client bbaaf009-516d-49f4-bfa2-128bc016cdab (at 192.168.203.56@tcp) reconnecting [ 2824.505741] Lustre: lustre-MDT0000: Client bbaaf009-516d-49f4-bfa2-128bc016cdab (at 192.168.203.56@tcp) reconnecting [ 2857.271848] Lustre: lustre-MDT0000: Client bbaaf009-516d-49f4-bfa2-128bc016cdab (at 192.168.203.56@tcp) reconnecting [ 2877.361910] Lustre: lustre-MDT0000: Client bbaaf009-516d-49f4-bfa2-128bc016cdab (at 192.168.203.56@tcp) reconnecting [ 2898.296150] Lustre: lustre-MDT0000: Client bbaaf009-516d-49f4-bfa2-128bc016cdab (at 192.168.203.56@tcp) reconnecting [ 2957.296937] Lustre: lustre-MDT0000: Client bbaaf009-516d-49f4-bfa2-128bc016cdab (at 192.168.203.56@tcp) reconnecting [ 2957.308349] Lustre: Skipped 1 previous similar message [ 3030.350016] Lustre: 7270:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 30372 > trans_max 3200 [ 3030.353236] Lustre: 7270:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 3030.356123] Lustre: 7270:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28148/0 [ 3030.359624] Lustre: 7270:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 3030.363123] Lustre: 7270:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 3030.366629] Lustre: 7270:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 3030.369272] CPU: 2 PID: 7270 Comm: mdt00_001 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 3030.373043] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 3030.376034] Call Trace: [ 3030.376808] [] dump_stack+0x19/0x1b [ 3030.378239] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 3030.381001] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 3030.383206] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 3030.385934] [] lod_trans_start+0x7f/0x2e0 [lod] [ 3030.388725] [] mdd_trans_start+0x14/0x20 [mdd] [ 3030.390834] [] mdd_unlink+0x5fb/0xe40 [mdd] [ 3030.394435] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 3030.396942] [] mdt_reint_unlink+0xbc7/0x15e0 [mdt] [ 3030.398758] [] mdt_reint_rec+0x87/0x240 [mdt] [ 3030.400607] [] mdt_reint_internal+0x74c/0xbc0 [mdt] [ 3030.402826] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 3030.407074] [] mdt_reint+0x67/0x150 [mdt] [ 3030.411800] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 3030.417945] [] ptlrpc_server_handle_request+0x281/0xce0 [ptlrpc] [ 3030.420701] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 3030.422632] [] ? put_prev_entity+0x31/0x400 [ 3030.434445] [] ? do_raw_spin_unlock+0x49/0x90 [ 3030.436395] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 3030.438042] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 3030.441807] [] kthread+0xe4/0xf0 [ 3030.443407] [] ? kthread_create_on_node+0x140/0x140 [ 3030.446997] [] ret_from_fork_nospec_begin+0x7/0x21 [ 3030.449537] [] ? kthread_create_on_node+0x140/0x140 [ 3033.000330] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 23:19:13 (1739506753) [ 3034.346134] Lustre: lustre-MDD0000: changelog on [ 3035.302225] Lustre: lustre-MDD0001: changelog on [ 3037.841961] Lustre: lustre-MDD0001: changelog off [ 3038.828605] Lustre: lustre-MDD0000: changelog off [ 3039.798245] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 23:19:20 (1739506760) [ 3039.944829] Lustre: lustre-OST0000: Client bbaaf009-516d-49f4-bfa2-128bc016cdab (at 192.168.203.56@tcp) reconnecting [ 3039.947409] Lustre: Skipped 1 previous similar message [ 3040.955998] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012db83000.ost_server_uuid 50 [ 3041.465288] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012db83000.ost_server_uuid in FULL state after 0 sec [ 3044.538307] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 23:19:25 (1739506765) [ 3047.559626] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 23:19:28 (1739506768) [ 3048.171020] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [ 3048.915956] Lustre: DEBUG MARKER: == sanity test 104d: runas -u 500 -g 500 lctl dl test ==== 23:19:29 (1739506769) [ 3051.631937] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 23:19:32 (1739506772) [ 3054.476975] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 23:19:35 (1739506775) [ 3057.256129] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 23:19:37 (1739506777) [ 3059.938243] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 23:19:40 (1739506780) [ 3072.885253] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 23:19:53 (1739506793) [ 3075.557957] Lustre: DEBUG MARKER: == sanity test 105f: Enqueue same range flocks =========== 23:19:56 (1739506796) [ 3079.381798] Lustre: DEBUG MARKER: == sanity test 105g: ldlm_lock_debug stack test ========== 23:20:00 (1739506800) [ 3084.551316] Lustre: DEBUG MARKER: == sanity test 105h: Flock functional verify ============= 23:20:05 (1739506805) [ 3088.040298] Lustre: DEBUG MARKER: == sanity test 105i: Flock deadlock verify =============== 23:20:08 (1739506808) [ 3096.010278] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 23:20:16 (1739506816) [ 3098.936894] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 23:20:19 (1739506819) [ 3103.371403] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 23:20:24 (1739506824) [ 3106.809632] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 23:20:27 (1739506827) [ 3257.678384] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 23:22:58 (1739506978) [ 3259.136760] Lustre: *** cfs_fail_loc=147, val=0*** [ 3263.015684] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 23:23:03 (1739506983) [ 3265.965996] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 23:23:06 (1739506986) [ 3269.415695] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 23:23:10 (1739506990) [ 3270.408012] Lustre: *** cfs_fail_loc=217, val=0*** [ 3270.410655] Lustre: Skipped 39 previous similar messages [ 3273.535072] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 3274.165245] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 3274.763569] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 23:23:15 (1739506995) [ 3277.887622] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 23:23:18 (1739506998) [ 3281.011132] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 23:23:21 (1739507001) [ 3281.898457] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3282.932270] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3284.988233] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3288.009333] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3292.017255] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3294.940144] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 23:23:35 (1739507015) [ 3304.321206] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 23:23:45 (1739507025) [ 3305.216872] Lustre: *** cfs_fail_loc=220, val=0*** [ 3305.218639] Lustre: Skipped 3 previous similar messages [ 3308.822910] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 23:23:49 (1739507029) [ 3319.374904] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 23:24:00 (1739507040) [ 3321.868212] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 23:24:02 (1739507042) [ 3324.520735] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 23:24:05 (1739507045) [ 3330.211881] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 23:24:11 (1739507051) [ 3333.082116] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 23:24:13 (1739507053) [ 3336.545218] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 23:24:17 (1739507057) [ 3340.102569] Lustre: DEBUG MARKER: == sanity test 119e: Basic tests of dio read and write at various sizes ========================================================== 23:24:20 (1739507060) [ 3357.254529] Lustre: DEBUG MARKER: == sanity test 119f: dio vs dio race ===================== 23:24:37 (1739507077) [ 3376.192122] Lustre: DEBUG MARKER: == sanity test 119g: dio vs buffered I/O race ============ 23:24:56 (1739507096) [ 3393.236053] Lustre: DEBUG MARKER: == sanity test 119h: basic tests of memory unaligned dio ========================================================== 23:25:13 (1739507113) [ 3405.349176] Lustre: DEBUG MARKER: SKIP: sanity test_119i skipping ALWAYS excluded test 119i [ 3406.111979] Lustre: DEBUG MARKER: == sanity test 119j: basic tests of hybrid IO switching == 23:25:26 (1739507126) [ 3406.725096] Lustre: DEBUG MARKER: SKIP: sanity test_119j needs kernel > 4.5.0 for ki_flags support [ 3407.419131] Lustre: DEBUG MARKER: == sanity test 119m: Test DIO readv/writev: exercise iter duplication ========================================================== 23:25:28 (1739507128) [ 3409.925058] Lustre: DEBUG MARKER: == sanity test 119n: Test Unaligned DIO readv() and writev() with unpatched ZFS ========================================================== 23:25:30 (1739507130) [ 3410.496239] Lustre: DEBUG MARKER: SKIP: sanity test_119n need ZFS server without unaligned_dio support [ 3411.243926] Lustre: DEBUG MARKER: == sanity test 119o: Test Unaligned DIO readv() and writev() with unpatched servers ========================================================== 23:25:31 (1739507131) [ 3411.904719] Lustre: DEBUG MARKER: SKIP: sanity test_119o need ldiskfs without unaligned_dio support. [ 3412.682358] Lustre: DEBUG MARKER: == sanity test 119p: Test Unaligned DIO readv() and writev() with patched servers ========================================================== 23:25:33 (1739507133) [ 3416.008759] Lustre: DEBUG MARKER: == sanity test 119q: Test patchded Unaligned DIO readv() and writev() ========================================================== 23:25:36 (1739507136) [ 3422.183410] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 23:25:42 (1739507142) [ 3426.506825] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 23:25:47 (1739507147) [ 3430.408585] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 23:25:51 (1739507151) [ 3434.911554] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 23:25:55 (1739507155) [ 3438.641898] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 23:25:59 (1739507159) [ 3451.349912] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 23:26:11 (1739507171) [ 3464.019533] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 23:26:24 (1739507184) [ 3680.696806] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 23:30:01 (1739507401) [ 3683.244582] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 23:30:04 (1739507404) [ 3687.361146] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 3688.385743] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 3708.016592] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 12 sec [ 3711.721280] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 3 sec [ 3776.874136] Lustre: DEBUG MARKER: ls -l 4000 files without statahead: 45 sec [ 3786.937077] Lustre: DEBUG MARKER: ls -l 4000 files with statahead: 8 sec [ 3787.545789] Lustre: DEBUG MARKER: ls -l done [ 3823.433778] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity: 36 seconds [ 3826.957162] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 23:32:27 (1739507547) [ 3827.773285] Lustre: DEBUG MARKER: SKIP: sanity test_123ab Test must be statx() syscall supported [ 3828.635306] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 23:32:29 (1739507549) [ 3829.366804] Lustre: DEBUG MARKER: SKIP: sanity test_123ac Test must be statx() syscall supported [ 3830.192369] Lustre: DEBUG MARKER: == sanity test 123ad: Verify batching statahead works correctly ========================================================== 23:32:30 (1739507550) [ 3834.806989] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 3836.100575] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 0 sec [ 3854.757146] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 11 sec [ 3859.553324] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 4 sec [ 3932.253402] Lustre: DEBUG MARKER: ls -l 4000 files without statahead: 53 sec [ 3949.794491] Lustre: DEBUG MARKER: ls -l 4000 files with statahead: 15 sec [ 3950.780860] Lustre: DEBUG MARKER: ls -l done [ 3988.735893] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 37 seconds [ 4278.048681] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 4279.367303] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 0 sec [ 4298.893960] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 12 sec [ 4302.413338] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 2 sec [ 4366.540803] Lustre: DEBUG MARKER: ls -l 4000 files without statahead: 45 sec [ 4374.783360] Lustre: DEBUG MARKER: ls -l 4000 files with statahead: 7 sec [ 4375.418949] Lustre: DEBUG MARKER: ls -l done [ 4410.239561] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 34 seconds [ 4613.473640] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 23:45:34 (1739508334) [ 4624.907854] Lustre: DEBUG MARKER: ls done [ 4635.719007] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 23:45:56 (1739508356) [ 4640.492279] Lustre: DEBUG MARKER: == sanity test 123d: Statahead on striped directories works correctly ========================================================== 23:46:01 (1739508361) [ 4648.571276] Lustre: DEBUG MARKER: == sanity test 123e: statahead with large wide striping == 23:46:09 (1739508369) [ 4742.228148] Lustre: DEBUG MARKER: == sanity test 123f: Retry mechanism with large wide striping files ========================================================== 23:47:42 (1739508462) [ 4786.405287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4786.426643] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4890.710952] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4891.294255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4960.484919] Lustre: 9621:0:(mdt_handler.c:4519:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 5007.863300] Lustre: lustre-MDT0000: Client 7ebeb6f3-48f4-4f13-a9cd-05805c7eecff (at 192.168.203.56@tcp) reconnecting [ 5045.773778] Lustre: 9822:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 26472 > trans_max 3200 [ 5045.779289] Lustre: 9822:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 5045.784617] Lustre: 9822:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1743/1743/0, xattr_set: 2614/24508/0 [ 5045.787588] Lustre: 9822:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 5045.797250] Lustre: 9822:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 5045.801571] Lustre: 9822:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 5045.810264] CPU: 3 PID: 9822 Comm: mdt00_007 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 5045.815026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 5045.818808] Call Trace: [ 5045.820228] [] dump_stack+0x19/0x1b [ 5045.821894] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 5045.824924] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 5045.828598] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 5045.831031] [] lod_trans_start+0x7f/0x2e0 [lod] [ 5045.833328] [] mdd_trans_start+0x14/0x20 [mdd] [ 5045.836026] [] mdd_unlink+0x5fb/0xe40 [mdd] [ 5045.837753] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 5045.839802] [] mdt_reint_unlink+0xbc7/0x15e0 [mdt] [ 5045.842839] [] mdt_reint_rec+0x87/0x240 [mdt] [ 5045.845143] [] mdt_reint_internal+0x74c/0xbc0 [mdt] [ 5045.847000] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 5045.850474] [] mdt_reint+0x67/0x150 [mdt] [ 5045.852272] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 5045.855486] [] ptlrpc_server_handle_request+0x281/0xce0 [ptlrpc] [ 5045.858773] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 5045.860897] [] ? do_raw_spin_unlock+0x49/0x90 [ 5045.863420] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 5045.866635] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 5045.870290] [] kthread+0xe4/0xf0 [ 5045.871857] [] ? kthread_create_on_node+0x140/0x140 [ 5045.874002] [] ret_from_fork_nospec_begin+0x7/0x21 [ 5045.876187] [] ? kthread_create_on_node+0x140/0x140 [ 5105.943030] Lustre: 7270:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 30372 > trans_max 3200 [ 5105.951171] Lustre: 7270:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 5105.956445] Lustre: 7270:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28148/0 [ 5105.963866] Lustre: 7270:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 5105.970775] Lustre: 7270:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 5105.976295] Lustre: 7270:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 5105.980574] CPU: 1 PID: 7270 Comm: mdt00_001 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 5105.986021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 5105.991066] Call Trace: [ 5105.996275] [] dump_stack+0x19/0x1b [ 5106.000216] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 5106.007854] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 5106.015325] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 5106.019345] [] lod_trans_start+0x7f/0x2e0 [lod] [ 5106.023604] [] mdd_trans_start+0x14/0x20 [mdd] [ 5106.027182] [] mdd_unlink+0x5fb/0xe40 [mdd] [ 5106.036085] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 5106.040466] [] mdt_reint_unlink+0xbc7/0x15e0 [mdt] [ 5106.042623] [] mdt_reint_rec+0x87/0x240 [mdt] [ 5106.047777] [] mdt_reint_internal+0x74c/0xbc0 [mdt] [ 5106.049842] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 5106.053817] [] mdt_reint+0x67/0x150 [mdt] [ 5106.055709] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 5106.058023] [] ptlrpc_server_handle_request+0x281/0xce0 [ptlrpc] [ 5106.060565] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 5106.062609] [] ? put_prev_entity+0x31/0x400 [ 5106.064417] [] ? do_raw_spin_unlock+0x49/0x90 [ 5106.066338] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 5106.068330] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 5106.070421] [] kthread+0xe4/0xf0 [ 5106.072042] [] ? kthread_create_on_node+0x140/0x140 [ 5106.074349] [] ret_from_fork_nospec_begin+0x7/0x21 [ 5106.076775] [] ? kthread_create_on_node+0x140/0x140 [ 5109.266659] Lustre: DEBUG MARKER: == sanity test 123g: Test for stat-ahead advise ========== 23:53:49 (1739508829) [ 5127.290594] Lustre: DEBUG MARKER: == sanity test 123h: Verify statahead work with the fname pattern via du ========================================================== 23:54:07 (1739508847) [ 5945.377850] Lustre: DEBUG MARKER: == sanity test 123i: Verify statahead work with the fname indexing pattern ========================================================== 00:07:45 (1739509665) [ 6036.973571] Lustre: DEBUG MARKER: == sanity test 123j: -ENOENT error from batched statahead be handled correctly ========================================================== 00:09:17 (1739509757) [ 6041.713171] Lustre: DEBUG MARKER: == sanity test 123k: Verify statahead work with mdtest shared stat() mode ========================================================== 00:09:22 (1739509762) [ 6042.458020] Lustre: DEBUG MARKER: SKIP: sanity test_123k mdtest not found [ 6043.499834] Lustre: DEBUG MARKER: == sanity test 123l: Avoid panic when revalidate a local cached entry ========================================================== 00:09:24 (1739509764) [ 6117.373741] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 00:10:37 (1739509837) [ 6118.174224] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 6137.775766] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff880095542000 [ 6138.474837] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff880095542000 [ 6139.242366] Lustre: DEBUG MARKER: LRU=1004 [ 6140.034967] Lustre: DEBUG MARKER: LIMIT=46624 [ 6140.682390] Lustre: DEBUG MARKER: LVF=5572500 [ 6141.390359] Lustre: DEBUG MARKER: OLD_LVF=100 [ 6142.126061] Lustre: DEBUG MARKER: Sleep 50 sec [ 6193.125072] Lustre: DEBUG MARKER: Dropped 425 locks in 50s [ 6193.939997] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 6206.688323] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 00:12:06 (1739509926) [ 6253.685449] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 6332.394613] Lustre: DEBUG MARKER: ls -la time: 78 seconds [ 6333.223190] Lustre: DEBUG MARKER: lru_size = 400 [ 6419.895402] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 6457.988858] Lustre: DEBUG MARKER: ls -la time: 36 seconds [ 6458.729898] Lustre: DEBUG MARKER: lru_size = 4052 [ 6459.441682] Lustre: DEBUG MARKER: ls -la is 53% faster with lru resize enabled [ 6491.604213] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 00:16:52 (1739510212) [ 6519.036956] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize disabled ========================================================== 00:17:19 (1739510239) [ 6545.687024] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 00:17:46 (1739510266) [ 6549.919071] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 00:17:50 (1739510270) [ 6553.645178] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 00:17:54 (1739510274) [ 6557.892807] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 00:17:58 (1739510278) [ 6561.499032] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [ 6580.128570] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 00:18:20 (1739510300) [ 6583.239941] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 00:18:23 (1739510303) [ 6591.449059] Lustre: 8793:0:(osd_handler.c:600:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513463, FID: [0x20000040c:0x82e0:0x0]) is approaching max size limit [ 6597.190499] Lustre: 8793:0:(osd_handler.c:596:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513463, FID: [0x20000040c:0x82e0:0x0]) has reached max size limit [ 6612.183426] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 00:18:52 (1739510332) [ 6615.825545] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 00:18:56 (1739510336) [ 6619.658677] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 00:19:00 (1739510340) [ 6622.914390] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 00:19:03 (1739510343) [ 6623.502716] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [ 6624.290444] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 00:19:04 (1739510344) [ 6655.174827] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 00:19:35 (1739510375) [ 6658.031441] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 00:19:38 (1739510378) [ 6673.666498] Lustre: 5025:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 6372 > trans_max 3200 [ 6673.670205] Lustre: 5025:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 6673.673851] Lustre: 5025:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 403/403/0, xattr_set: 604/5748/0 [ 6673.678319] Lustre: 5025:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 6673.683507] Lustre: 5025:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 6673.688187] Lustre: 5025:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 6673.692074] CPU: 0 PID: 5025 Comm: mdt00_010 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 6673.697619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 6673.700713] Call Trace: [ 6673.702229] [] dump_stack+0x19/0x1b [ 6673.705155] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 6673.708416] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 6673.711199] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 6673.713591] [] lod_trans_start+0x7f/0x2e0 [lod] [ 6673.715506] [] mdd_trans_start+0x14/0x20 [mdd] [ 6673.717525] [] mdd_unlink+0x5fb/0xe40 [mdd] [ 6673.719591] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 6673.722700] [] mdt_reint_unlink+0xbc7/0x15e0 [mdt] [ 6673.728330] [] mdt_reint_rec+0x87/0x240 [mdt] [ 6673.732225] [] mdt_reint_internal+0x74c/0xbc0 [mdt] [ 6673.736492] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 6673.738608] [] mdt_reint+0x67/0x150 [mdt] [ 6673.741406] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 6673.743847] [] ptlrpc_server_handle_request+0x281/0xce0 [ptlrpc] [ 6673.750052] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 6673.754234] [] ? do_raw_spin_unlock+0x49/0x90 [ 6673.756310] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 6673.759598] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 6673.764134] [] kthread+0xe4/0xf0 [ 6673.766043] [] ? kthread_create_on_node+0x140/0x140 [ 6673.768212] [] ret_from_fork_nospec_begin+0x7/0x21 [ 6673.771146] [] ? kthread_create_on_node+0x140/0x140 [ 6676.271908] Lustre: DEBUG MARKER: == sanity test 130h: FIEMAP deadlock ===================== 00:19:56 (1739510396) [ 6685.477539] Lustre: DEBUG MARKER: == sanity test 130i: FIEMAP (DoM file) =================== 00:20:06 (1739510406) [ 6706.417353] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 00:20:27 (1739510427) [ 6710.026018] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 00:20:30 (1739510430) [ 6713.303668] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 00:20:33 (1739510433) [ 6716.039158] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 00:20:36 (1739510436) [ 6718.797692] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 00:20:39 (1739510439) [ 6721.554643] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 00:20:42 (1739510442) [ 6730.290699] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 00:20:51 (1739510451) [ 6735.626384] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 00:20:56 (1739510456) [ 6755.571736] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 00:21:16 (1739510476) [ 6771.140761] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 00:21:31 (1739510491) [ 6777.293768] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 00:21:37 (1739510497) [ 6786.674476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6786.675384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6786.686334] Lustre: Skipped 2 previous similar messages [ 6788.498492] Lustre: server umount lustre-MDT0000 complete [ 6791.682606] LustreError: 9822:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6791.694015] LustreError: 9822:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 5 previous similar messages [ 6792.142808] LustreError: 10073:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1739510512 with bad export cookie 18312317060492169405 [ 6792.148249] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6792.153326] LustreError: 10073:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6792.317292] mdt00_005 (8801) used greatest stack depth: 9824 bytes left [ 6792.396334] Lustre: server umount lustre-MDT0001 complete [ 6806.806155] Lustre: server umount lustre-OST0000 complete [ 6821.747934] Lustre: server umount lustre-OST0001 complete [ 6824.640708] device-mapper: core: cleaned up [ 6828.634806] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 6829.559920] Key type lgssc unregistered [ 6829.675739] LNet: 28521:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6830.685578] LNet: Removed LNI 192.168.203.156@tcp [ 6836.664125] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 6836.819592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6836.835978] alg: No test for adler32 (adler32-zlib) [ 6837.687168] Lustre: Lustre: Build Version: 2.16.52_26_g7793934 [ 6837.765038] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 6837.767328] LNet: Accept secure, port 988 [ 6839.315355] Key type lgssc registered [ 6839.678108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6845.683107] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 6848.199951] device-mapper: uevent: version 1.0.3 [ 6848.205232] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 6851.604008] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6851.621220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6852.903258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6854.185957] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [ 6859.485382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6859.552465] Lustre: 31799:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6859.747959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6859.805740] mount.lustre (31799) used greatest stack depth: 9520 bytes left [ 6861.419753] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [ 6862.830864] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 6867.250080] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6867.257991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6867.367127] Lustre: 464:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6867.466503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6869.466410] LustreError: 469:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6869.478980] LustreError: 469:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 2 previous similar messages [ 6869.489079] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:11627 to 0x280000400:11649) [ 6869.687575] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [ 6874.466996] LustreError: 468:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6874.534931] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6874.543110] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6874.597373] Lustre: 1694:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6874.673767] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6876.635227] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [ 6878.678367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:22564 to 0x280000403:22657) [ 6878.679203] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:22948 to 0x2c0000403:23041) [ 6878.697170] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:11460 to 0x2c0000400:11489) [ 6885.133284] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6886.623387] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 6894.994323] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 00:23:35 (1739510615) [ 6920.845104] LNet: 4638:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 6920.852256] ------------[ cut here ]------------ [ 6920.853902] WARNING: CPU: 0 PID: 4638 at mm/page_alloc.c:3222 __alloc_pages_nodemask+0x53f/0xca0 [ 6920.856860] Modules linked in: dm_flakey dm_mod lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc_t10dif crct10dif_generic rpcsec_gss_krb5 sb_edac edac_core iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel squashfs lrw gf128mul glue_helper ablk_helper cryptd i2c_piix4 i2c_core pcspkr binfmt_misc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi crct10dif_pclmul ata_piix crct10dif_common crc32c_intel serio_raw libata [last unloaded: libcfs] [ 6920.881791] CPU: 0 PID: 4638 Comm: badarea_io Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 6920.886866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 6920.891151] Call Trace: [ 6920.892370] [] dump_stack+0x19/0x1b [ 6920.893952] [] __warn+0xd8/0x100 [ 6920.895213] [] warn_slowpath_null+0x1d/0x20 [ 6920.897864] [] __alloc_pages_nodemask+0x53f/0xca0 [ 6920.901072] [] alloc_pages_current+0x98/0x110 [ 6920.902802] [] kmalloc_order+0x1f/0x70 [ 6920.904584] [] kmalloc_order_trace+0x26/0xc0 [ 6920.906588] [] __kmalloc_track_caller+0x30e/0x330 [ 6920.908778] [] ? proc_dobitmasks+0xcb/0x140 [libcfs] [ 6920.911279] [] ? __enqueue_entity+0x78/0x80 [ 6920.913499] [] memdup_user_nul+0x26/0x90 [ 6920.917316] [] proc_dobitmasks+0xcb/0x140 [libcfs] [ 6920.919906] [] lnet_debugfs_write+0x43/0x70 [libcfs] [ 6920.922981] [] vfs_write+0xd6/0x230 [ 6920.925641] [] ? __schedule+0x32a/0x7d0 [ 6920.927495] [] SyS_write+0x7f/0xf0 [ 6920.931062] [] system_call_fastpath+0x1f/0x24 [ 6920.933043] ---[ end trace 0c5c515540ee5654 ]--- [ 6920.939937] Lustre: DEBUG MARKER:  [ 6951.008839] LNet: 6445:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 6951.012522] LNet: 6445:0:(debug.c:388:cfs_str2mask()) Skipped 3 previous similar messages [ 6951.037129] Lustre: DEBUG MARKER:  [ 6951.038125] Lustre: DEBUG MARKER:  [ 6953.793586] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6953.794230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6953.802664] Lustre: Skipped 2 previous similar messages [ 6958.804699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6958.811228] Lustre: Skipped 6 previous similar messages [ 6959.590111] Lustre: server umount lustre-MDT0000 complete [ 6963.493451] LustreError: 30715:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1739510683 with bad export cookie 17113498151192185414 [ 6963.498350] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6963.502604] LustreError: 30715:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6963.682190] Lustre: server umount lustre-MDT0001 complete [ 6977.622396] Lustre: server umount lustre-OST0000 complete [ 6980.600794] Lustre: 28940:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1739510684/real 1739510684] req@ffff880096775500 x1824009021980544/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1739510700 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 6980.624357] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6980.636138] Lustre: Skipped 1 previous similar message [ 6981.835685] Lustre: server umount lustre-OST0001 complete [ 6985.023722] device-mapper: core: cleaned up [ 6989.500828] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 6990.726741] Key type lgssc unregistered [ 6990.851277] LNet: 9431:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6991.859365] LNet: Removed LNI 192.168.203.156@tcp [ 6997.709643] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 6997.966069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6997.972095] alg: No test for adler32 (adler32-zlib) [ 6998.822404] Lustre: Lustre: Build Version: 2.16.52_26_g7793934 [ 6998.913796] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 6998.915892] LNet: Accept secure, port 988 [ 7000.465561] Key type lgssc registered [ 7000.836825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7006.531492] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 7008.809799] device-mapper: uevent: version 1.0.3 [ 7008.812826] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 7012.041982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7012.053957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7013.317433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7014.301014] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [ 7018.281296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7018.335053] Lustre: 12711:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7018.485375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7018.521930] mount.lustre (12711) used greatest stack depth: 9504 bytes left [ 7019.736491] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [ 7020.907304] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 7024.456063] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7024.464784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7024.607119] Lustre: 13839:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7024.701141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7026.795109] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [ 7028.698690] LustreError: 14545:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.705640] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:11627 to 0x280000400:11681) [ 7028.714462] LustreError: 14545:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 2 previous similar messages [ 7031.557635] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 7031.565086] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7031.625142] Lustre: 15034:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7031.697912] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7033.737609] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [ 7036.713218] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:11460 to 0x2c0000400:11521) [ 7038.721263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:22564 to 0x280000403:22689) [ 7038.722058] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:22948 to 0x2c0000403:23073) [ 7043.343422] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7044.885321] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 7052.805744] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 00:26:13 (1739510773) [ 7526.258915] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 00:34:06 (1739511246) [ 7534.987003] Lustre: *** cfs_fail_loc=327, val=500*** [ 7556.599778] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 00:34:37 (1739511277) [ 7559.857694] Lustre: *** cfs_fail_loc=328, val=500*** [ 7559.859940] Lustre: Skipped 1387 previous similar messages [ 7560.860349] Lustre: *** cfs_fail_loc=328, val=500*** [ 7560.862238] Lustre: Skipped 141 previous similar messages [ 7561.675059] Lustre: 3067:0:(ldlm_lockd.c:1365:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8800750f1880 x1824009188607488/t0(0) o101->933ab7a1-8bbf-4db9-a337-e7691e370a91@192.168.203.56@tcp:352/0 lens 648/0 e 0 to 0 dl 1739511292 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7562.859840] Lustre: *** cfs_fail_loc=328, val=500*** [ 7562.862142] Lustre: Skipped 127 previous similar messages [ 7563.627819] Lustre: 3067:0:(ldlm_lockd.c:1365:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8800750f2680 x1824009188672256/t0(0) o101->933ab7a1-8bbf-4db9-a337-e7691e370a91@192.168.203.56@tcp:354/0 lens 648/0 e 0 to 0 dl 1739511294 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7564.663944] Lustre: 15725:0:(ldlm_lockd.c:1365:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff880067b8d880 x1824009188673024/t0(0) o101->933ab7a1-8bbf-4db9-a337-e7691e370a91@192.168.203.56@tcp:355/0 lens 648/0 e 0 to 0 dl 1739511295 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7566.729115] Lustre: 13674:0:(ldlm_lockd.c:1365:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8800750f3100 x1824009188674560/t0(0) o101->933ab7a1-8bbf-4db9-a337-e7691e370a91@192.168.203.56@tcp:357/0 lens 648/0 e 0 to 0 dl 1739511297 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7566.741510] Lustre: 13674:0:(ldlm_lockd.c:1365:ldlm_handle_enqueue()) Skipped 1 previous similar message [ 7567.761991] Lustre: *** cfs_fail_loc=328, val=500*** [ 7567.763737] Lustre: Skipped 110 previous similar messages [ 7571.262581] Lustre: 11640:0:(ldlm_lockd.c:1365:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff88007ea67100 x1824009188708864/t0(0) o101->933ab7a1-8bbf-4db9-a337-e7691e370a91@192.168.203.56@tcp:362/0 lens 648/0 e 0 to 0 dl 1739511302 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7571.273342] Lustre: 11640:0:(ldlm_lockd.c:1365:ldlm_handle_enqueue()) Skipped 3 previous similar messages [ 7576.464630] Lustre: *** cfs_fail_loc=328, val=500*** [ 7576.466132] Lustre: Skipped 94 previous similar messages [ 7579.623718] LustreError: 23254:0:(ldlm_resource.c:112:seq_watermark_write()) Failed to set lock_reclaim_threshold_mb, rc = -22. [ 7586.718882] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [ 7587.666463] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [ 7588.574089] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 00:35:09 (1739511309) [ 7611.590077] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 00:35:32 (1739511332) [ 7631.905985] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 00:35:52 (1739511352) [ 7647.983621] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 00:36:08 (1739511368) [ 7666.985256] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 00:36:27 (1739511387) [ 7678.624322] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 00:36:38 (1739511398) [ 7692.762921] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 00:36:53 (1739511413) [ 7708.890746] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 00:37:09 (1739511429) [ 7718.656538] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 00:37:19 (1739511439) [ 7737.183539] Lustre: DEBUG MARKER: == sanity test 150h: Verify extend fallocate updates the file size ========================================================== 00:37:37 (1739511457) [ 7738.124589] Lustre: DEBUG MARKER: SKIP: sanity test_150h Test must be statx() syscall supported [ 7738.767913] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 00:37:39 (1739511459) [ 7746.730433] bash (32687): drop_caches: 1 [ 7752.896359] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 00:37:53 (1739511473) [ 7756.107221] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 00:37:56 (1739511476) [ 7759.139345] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 00:37:59 (1739511479) [ 7762.403250] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 00:38:03 (1739511483) [ 7765.720261] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 00:38:06 (1739511486) [ 7766.512179] LustreError: 11640:0:(fld_handler.c:244:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 7770.580008] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 00:38:11 (1739511491) [ 7771.237376] LustreError: 11638:0:(fld_handler.c:244:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 7771.242541] LustreError: 11638:0:(fld_handler.c:244:fld_server_lookup()) Skipped 3 previous similar messages [ 7775.068022] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 00:38:15 (1739511495) [ 7777.877393] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 00:38:18 (1739511498) [ 7781.156183] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 00:38:21 (1739511501) [ 7783.814383] Lustre: DEBUG MARKER: == sanity test 154ea: .lustre is not returned by readdir (2) ========================================================== 00:38:24 (1739511504) [ 7817.919259] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 00:38:58 (1739511538) [ 7822.103703] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 00:39:02 (1739511542) [ 8258.118224] Lustre: DEBUG MARKER: == sanity test 154h: Verify interactive path2fid ========= 00:46:18 (1739511978) [ 8261.901624] Lustre: DEBUG MARKER: == sanity test 154i: fid2path for path longer than PATH_MAX ========================================================== 00:46:22 (1739511982) [ 8286.690958] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 00:46:47 (1739512007) [ 8294.984285] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 00:46:55 (1739512015) [ 8302.235242] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 00:47:02 (1739512022) [ 8309.868542] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 00:47:10 (1739512030) [ 8318.176296] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 00:47:18 (1739512038) [ 8349.732182] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 00:47:50 (1739512070) [ 8379.584762] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 00:48:20 (1739512100) [ 8410.061676] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 00:48:50 (1739512130) [ 8441.228383] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 00:49:21 (1739512161) [ 8445.969912] Lustre: DEBUG MARKER: Turn on read and write cache [ 8447.969249] Lustre: DEBUG MARKER: Write data and read it back. [ 8448.656380] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 8450.613051] Lustre: DEBUG MARKER: cache hits: before: 65581, after: 65584 [ 8451.326396] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8452.670293] Lustre: DEBUG MARKER: cache hits:: before: 65584, after: 65587 [ 8453.241020] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [ 8454.883521] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8456.901030] Lustre: DEBUG MARKER: cache hits:: before: 65587, after: 65590 [ 8457.696210] Lustre: DEBUG MARKER: Write data and read it back. [ 8458.309186] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 8460.412992] Lustre: DEBUG MARKER: cache hits:: before: 65590, after: 65593 [ 8461.283339] Lustre: DEBUG MARKER: Turn off read and write cache [ 8471.213678] Lustre: DEBUG MARKER: Write data and read it back [ 8472.186581] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 8474.570385] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65593 [ 8475.792017] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [ 8478.045310] Lustre: DEBUG MARKER: Write data and read it back [ 8478.964057] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 8481.163050] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65593 [ 8481.996317] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8484.413184] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65596 [ 8489.269940] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 00:50:09 (1739512209) [ 8491.017028] Lustre: lustre-MDD0000: changelog on [ 8492.575143] Lustre: lustre-MDD0001: changelog on [ 8502.100804] Lustre: Failing over lustre-MDT0000 [ 8502.370438] Lustre: server umount lustre-MDT0000 complete [ 8504.461364] LustreError: 3431:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8506.114378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8506.119978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8506.127992] LustreError: 13853:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8507.135390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8507.236685] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8507.512333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8507.538967] Lustre: lustre-MDD0000: changelog on [ 8507.553116] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8509.460348] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8509.939080] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [ 8512.523258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 8512.783299] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8512.829974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:23525 to 0x280000403:23553) [ 8512.832504] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23913 to 0x2c0000403:23937) [ 8513.320276] Lustre: lustre-MDD0000: changelog off [ 8514.343212] Lustre: lustre-MDD0001: changelog off [ 8521.776515] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 00:50:42 (1739512242) [ 8523.150565] Lustre: lustre-MDD0000: changelog on [ 8528.012329] Lustre: lustre-MDD0001: changelog off [ 8530.185585] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 00:50:50 (1739512250) [ 8531.595050] Lustre: lustre-MDD0000: changelog on [ 8531.597181] Lustre: Skipped 1 previous similar message [ 8538.301670] Lustre: lustre-MDD0001: changelog off [ 8538.303126] Lustre: Skipped 1 previous similar message [ 8541.358713] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 00:51:01 (1739512261) [ 8542.861663] Lustre: lustre-MDD0000: changelog on [ 8542.863300] Lustre: Skipped 1 previous similar message [ 8548.951140] Lustre: lustre-MDD0001: changelog off [ 8548.953436] Lustre: Skipped 1 previous similar message [ 8552.154976] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 00:51:12 (1739512272) [ 8559.845667] Lustre: lustre-MDD0001: changelog off [ 8559.848681] Lustre: Skipped 1 previous similar message [ 8562.388236] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 00:51:23 (1739512283) [ 8563.872816] Lustre: lustre-MDD0000: changelog on [ 8563.875752] Lustre: Skipped 3 previous similar messages [ 8568.996885] Lustre: DEBUG MARKER: 1739512289: creating first dirs [ 8585.427677] Lustre: *** cfs_fail_loc=1313, val=3*** [ 8585.429336] Lustre: 11640:0:(mdd_dir.c:911:mdd_changelog_emrg_cleanup()) lustre-MDD0000: changelog has only 3 free catalog entries [ 8585.434332] Lustre: 11640:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 8585.441152] Lustre: 25824:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl7 idle for 19s with 4 unprocessed records [ 8597.282514] Lustre: lustre-MDD0001: changelog off [ 8597.284607] Lustre: Skipped 1 previous similar message [ 8600.349176] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 00:52:00 (1739512320) [ 8602.039148] Lustre: lustre-MDD0000: changelog on [ 8602.040994] Lustre: Skipped 1 previous similar message [ 8613.286273] Lustre: 3431:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 8613.290281] Lustre: 3431:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 8613.299149] Lustre: 29124:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl9 idle for 8s with 4 unprocessed records [ 8613.307295] Lustre: 29124:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 8627.139938] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 00:52:27 (1739512347) [ 8651.635939] Lustre: *** cfs_fail_loc=1316, val=0*** [ 8651.646863] Lustre: 11638:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0001: simulate starting changelog garbage collection [ 8651.653820] Lustre: 11638:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 8651.667775] Lustre: 32415:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0001: force deregister of changelog user cl11 idle for 19s with 3 unprocessed records [ 8651.675949] Lustre: 32415:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 8653.019819] Lustre: Failing over lustre-MDT0000 [ 8653.081912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8653.084503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8653.100450] Lustre: Skipped 3 previous similar messages [ 8653.107043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8653.234558] Lustre: Failing over lustre-MDT0001 [ 8654.786516] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 8657.769908] Lustre: server umount lustre-MDT0000 complete [ 8657.890021] Lustre: server umount lustre-MDT0001 complete [ 8662.376003] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8662.450494] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8662.600007] LustreError: 13853:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8662.601319] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8662.620615] LustreError: 13853:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 6 previous similar messages [ 8662.659536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8662.699913] Lustre: 1529:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 31 to index 32, being cleared now [ 8662.716593] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8664.104052] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [ 8667.667584] LustreError: 1540:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8667.675925] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 8667.678465] Lustre: Skipped 3 previous similar messages [ 8668.201027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8668.303053] Lustre: 2416:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8668.421997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 8668.530309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8668.556806] Lustre: lustre-MDD0001: changelog on [ 8668.560738] Lustre: Skipped 4 previous similar messages [ 8668.566728] Lustre: 2416:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0001 : orphan changelog records found, starting from index 15 to index 16, being cleared now [ 8668.580592] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8670.128984] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [ 8673.542321] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [ 8673.550669] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 8673.550670] Lustre: Skipped 1 previous similar message [ 8673.562842] Lustre: Skipped 1 previous similar message [ 8674.535986] Lustre: 9850:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1739512378/real 1739512378] req@ffff88007ffa7480 x1824009193509504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1739512394 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 8674.565349] Lustre: 9850:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8674.889068] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8674.917220] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:11686 to 0x280000400:11713) [ 8674.917433] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:11525 to 0x2c0000400:11553) [ 8675.274495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:23555 to 0x280000403:23585) [ 8675.274664] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23913 to 0x2c0000403:23969) [ 8683.638370] Lustre: lustre-MDD0001: changelog off [ 8683.639998] Lustre: Skipped 3 previous similar messages [ 8686.141057] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 00:53:26 (1739512406) [ 8691.350993] LustreError: 4983:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_race id 1315 sleeping [ 8693.569721] LustreError: 5153:0:(mdd_device.c:1746:mdd_changelog_user_register()) cfs_fail_race id 1315 waking [ 8693.586299] LustreError: 4983:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_fail_race id 1315 awake: rc=2771 [ 8694.454239] LustreError: 5406:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_fail_race id 1315 waking [ 8705.985184] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 00:53:46 (1739512426) [ 8719.136542] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 00:53:59 (1739512439) [ 8723.060564] LustreError: 1539:0:(mdd_dir.c:966:mdd_changelog_store()) cfs_fail_timeout id 15d sleeping for 3000ms [ 8726.070738] LustreError: 1539:0:(mdd_dir.c:966:mdd_changelog_store()) cfs_fail_timeout id 15d awake [ 8735.879363] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 00:54:16 (1739512456) [ 8746.821235] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 00:54:27 (1739512467) [ 8753.384915] LustreError: 8552:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_race id 15f sleeping [ 8755.391079] LustreError: 1539:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_fail_race id 15f waking [ 8755.395098] LustreError: 1539:0:(mdd_device.c:394:llog_changelog_cancel_cb()) Skipped 1 previous similar message [ 8755.402867] LustreError: 8552:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_fail_race id 15f awake: rc=2986 [ 8761.540942] Lustre: lustre-MDD0001: changelog off [ 8761.543922] Lustre: Skipped 11 previous similar messages [ 8764.123165] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 00:54:44 (1739512484) [ 9812.489383] LustreError: 1539:0:(mdd_device.c:408:llog_changelog_cancel_cb()) cfs_race id 16c sleeping [ 9814.490145] LustreError: 13815:0:(llog_osd.c:1136:llog_osd_next_block()) cfs_fail_race id 16c waking [ 9814.499968] LustreError: 1539:0:(mdd_device.c:408:llog_changelog_cancel_cb()) cfs_fail_race id 16c awake: rc=2991 [ 9821.560072] Lustre: lustre-MDD0001: changelog off [ 9821.563038] Lustre: Skipped 1 previous similar message [ 9824.306742] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 01:12:24 (1739513544) [ 9825.966690] Lustre: lustre-MDD0000: changelog on [ 9825.968960] Lustre: Skipped 14 previous similar messages [ 9827.957934] LustreError: 16228:0:(mdd_device.c:1700:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [ 9828.463593] Lustre: 16292:0:(mdd_device.c:1717:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [ 9829.018784] LustreError: 16357:0:(mdd_device.c:1709:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [ 9842.411301] Lustre: lustre-MDD0000: changelog off [ 9842.413156] Lustre: Skipped 1 previous similar message [ 9845.909581] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 01:12:46 (1739513566) [ 9847.252403] Lustre: lustre-MDD0000: changelog on [ 9847.254567] Lustre: Skipped 1 previous similar message [ 9850.674681] Lustre: Failing over lustre-MDT0000 [ 9850.873526] Lustre: server umount lustre-MDT0000 complete [ 9852.212150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9852.217109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9852.227849] Lustre: Skipped 3 previous similar messages [ 9852.230669] LustreError: 13115:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9852.237492] LustreError: 13115:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 2 previous similar messages [ 9854.014965] LustreError: 13815:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9855.425179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9855.506143] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9855.511911] LustreError: 13115:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9855.532717] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9855.539722] Lustre: Skipped 2 previous similar messages [ 9855.759018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9855.814194] Lustre: 19473:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 90155 to index 18446744073709551615, being cleared now [ 9855.829712] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9857.433772] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [ 9858.996059] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9860.756520] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 9860.759950] Lustre: Skipped 3 previous similar messages [ 9860.767219] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9860.777075] Lustre: Skipped 1 previous similar message [ 9860.816027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:23587 to 0x280000403:23617) [ 9860.816867] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23971 to 0x2c0000403:24001) [ 9867.385355] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 01:13:07 (1739513587) [ 9873.255140] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records * time ========================================================== 01:13:13 (1739513593) [ 9883.084234] Lustre: 13815:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 9883.090228] Lustre: 13815:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 9883.102009] Lustre: 22053:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl2 idle for 864008s with 500000004 unprocessed records [ 9883.110808] Lustre: 22053:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 9883.120296] Lustre: lustre-MDD0000: changelog off [ 9883.123261] Lustre: Skipped 3 previous similar messages [ 9892.358392] Lustre: DEBUG MARKER: == sanity test 160t: changelog garbage collect on lack of space ========================================================== 01:13:33 (1739513613) [ 9893.762926] Lustre: lustre-MDD0000: changelog on [ 9893.766147] Lustre: Skipped 4 previous similar messages [ 9924.831749] Lustre: *** cfs_fail_loc=18c, val=1211180*** [ 9924.834278] Lustre: Skipped 1 previous similar message [ 9924.835913] Lustre: 1540:0:(mdd_dir.c:881:mdd_changelog_is_space_safe()) lustre-MDD0000: changelog uses 21MB with 1MB space limit [ 9924.839298] Lustre: 1540:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 9924.842866] Lustre: 1540:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 9924.849322] Lustre: 24716:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl3-user1 idle for 31s with 7506 unprocessed records [ 9924.855631] Lustre: 24716:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 9935.499398] Lustre: DEBUG MARKER: == sanity test 160u: changelog rename record type name and sname strings are correct ========================================================== 01:14:16 (1739513656) [ 9945.767806] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 01:14:26 (1739513666) [ 9963.129944] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 01:14:43 (1739513683) [ 9980.688407] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 01:15:01 (1739513701) [ 9982.212211] Lustre: lustre-MDD0000: changelog on [ 9982.214604] Lustre: Skipped 3 previous similar messages [ 9988.727571] Lustre: lustre-MDD0001: changelog off [ 9988.729487] Lustre: Skipped 5 previous similar messages [ 9990.758255] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 01:15:11 (1739513711) [10000.266261] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 01:15:20 (1739513720) [10004.033712] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 01:15:24 (1739513724) [10008.349565] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 01:15:28 (1739513728) [10046.502618] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 01:16:07 (1739513767) [10053.185318] Lustre: Failing over lustre-OST0000 [10053.489378] Lustre: server umount lustre-OST0000 complete [10054.323610] LustreError: 22352:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10054.330587] LustreError: 22352:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 3 previous similar messages [10056.101347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10056.110255] Lustre: Skipped 1 previous similar message [10059.315193] LustreError: 22352:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10059.326653] LustreError: 22352:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 2 previous similar messages [10062.085275] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10062.090274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10062.204878] Lustre: 327:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [10062.337816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10062.352968] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10063.594071] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10064.352445] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.156@tcp (at 0@lo) [10064.353732] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10064.366686] Lustre: Skipped 3 previous similar messages [10064.805809] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [10066.840040] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 01:16:27 (1739513787) [10092.933273] Lustre: Failing over lustre-OST0000 [10092.991398] Lustre: server umount lustre-OST0000 complete [10094.391039] LustreError: 3435:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10094.401020] LustreError: 3435:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 2 previous similar messages [10094.417086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10094.420444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10094.431887] LustreError: Skipped 1 previous similar message [10097.198518] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10097.206261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10097.264595] Lustre: 2354:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [10097.352093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10097.360619] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10098.388484] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10098.704257] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10098.708956] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.156@tcp (at 0@lo) [10098.721061] Lustre: Skipped 1 previous similar message [10099.965355] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [10101.937070] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 01:17:02 (1739513822) [10121.095346] Lustre: Failing over lustre-OST0000 [10121.169451] Lustre: server umount lustre-OST0000 complete [10122.386111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10122.401088] Lustre: Skipped 1 previous similar message [10122.407284] LustreError: 3426:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10122.427309] LustreError: 3426:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 3 previous similar messages [10125.641512] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10125.654955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10125.729601] Lustre: 4353:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [10125.828456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10125.838796] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10127.388068] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10127.556040] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10127.556161] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.156@tcp (at 0@lo) [10127.556162] Lustre: Skipped 1 previous similar message [10128.544973] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [10130.517530] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 01:17:31 (1739513851) [10160.710068] Lustre: Failing over lustre-OST0000 [10160.820188] Lustre: server umount lustre-OST0000 complete [10160.902402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10160.907205] Lustre: Skipped 1 previous similar message [10160.910285] LustreError: 3421:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10160.928794] LustreError: 3421:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 2 previous similar messages [10165.428301] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10165.443007] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10165.563321] Lustre: 7394:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [10165.703259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10165.717488] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10167.205055] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10167.488544] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.156@tcp (at 0@lo) [10167.488648] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10167.503427] Lustre: Skipped 1 previous similar message [10168.471038] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [10170.375113] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 01:18:10 (1739513890) [10186.408063] Lustre: Failing over lustre-OST0000 [10187.541624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10187.544894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10187.563083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10187.563084] Lustre: Skipped 1 previous similar message [10187.570962] Lustre: Skipped 2 previous similar messages [10188.472539] Lustre: server umount lustre-OST0000 complete [10193.590737] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10193.600274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10193.667135] Lustre: 9423:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [10193.801230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10193.816737] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10194.529594] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10195.627691] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10195.628644] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.156@tcp (at 0@lo) [10195.628646] Lustre: Skipped 1 previous similar message [10196.443143] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [10198.742750] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 01:18:39 (1739513919) [10205.415030] Lustre: Failing over lustre-OST0000 [10205.665016] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10205.668138] LustreError: Skipped 1 previous similar message [10205.758955] Lustre: server umount lustre-OST0000 complete [10216.859048] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10216.868097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10216.983593] Lustre: 11349:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [10217.093284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10217.108932] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10218.149179] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10218.917689] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10218.924520] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.156@tcp (at 0@lo) [10218.928790] Lustre: Skipped 1 previous similar message [10220.559680] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [10222.997614] Lustre: DEBUG MARKER: == sanity test 165g: ofd_access_log_reader --keepalive works ========================================================== 01:19:03 (1739513943) [10265.136771] Lustre: Failing over lustre-OST0000 [10265.193126] Lustre: server umount lustre-OST0000 complete [10267.171164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10267.172946] LustreError: 3426:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10267.172948] LustreError: 3426:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 12 previous similar messages [10267.211963] Lustre: Skipped 3 previous similar messages [10274.387994] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10274.398447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10274.505187] Lustre: 13435:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [10274.654553] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10274.674994] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10275.906978] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10275.938751] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10275.945817] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.156@tcp (at 0@lo) [10275.945818] Lustre: Skipped 1 previous similar message [10277.336050] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [10279.473919] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 01:19:59 (1739513999) [10280.388327] Lustre: DEBUG MARKER: creating a 10 Mb file [10308.281096] Lustre: DEBUG MARKER: starting reads [10309.164396] Lustre: DEBUG MARKER: truncating the file [10310.219147] Lustre: DEBUG MARKER: killing dd [10310.899995] Lustre: DEBUG MARKER: removing the temporary file [10314.040999] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log =============================================================================== 01:20:34 (1739514034) [10318.219412] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 01:20:38 (1739514038) [10325.168262] Lustre: DEBUG MARKER: == sanity test 172: manual device removal with lctl cleanup/detach ================================================================ 01:20:45 (1739514045) [10332.495547] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 01:20:53 (1739514053) [10333.352398] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [10334.323063] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 01:20:54 (1739514054) [10335.749134] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_module obdecho/obdecho [10349.392718] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 01:21:09 (1739514069) [10351.157037] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_module obdecho/obdecho [10351.282489] Lustre: Echo OBD driver; http://www.lustre.org/ [10362.764671] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 01:21:23 (1739514083) [10412.235098] Lustre: DEBUG MARKER: == sanity test 182a: Test parallel modify metadata operations from mdc ========================================================== 01:22:12 (1739514132) [10443.924645] Lustre: DEBUG MARKER: == sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 01:22:43 (1739514163) [10858.030090] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 01:29:38 (1739514578) [10858.827756] Lustre: *** cfs_fail_loc=148, val=0*** [10863.228840] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 01:29:43 (1739514583) [10869.812499] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 01:29:50 (1739514590) [10873.822581] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 01:29:54 (1739514594) [10896.974508] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 01:30:17 (1739514617) [10901.770879] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 01:30:22 (1739514622) [10907.149373] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 01:30:27 (1739514627) [10911.184577] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 01:30:31 (1739514631) [10916.660370] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 01:30:37 (1739514637) [10925.127233] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 01:30:45 (1739514645) [10930.765690] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 01:30:51 (1739514651) [10935.835648] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 01:30:56 (1739514656) [10958.147015] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 01:31:18 (1739514678) [10961.490937] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 01:31:22 (1739514682) [10965.095105] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 01:31:25 (1739514685) [10968.557725] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 01:31:29 (1739514689) [10971.877591] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 01:31:32 (1739514692) [10975.237454] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 01:31:35 (1739514695) [10978.655168] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 01:31:39 (1739514699) [10982.207535] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 01:31:42 (1739514702) [10985.606111] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 01:31:46 (1739514706) [10989.953678] Lustre: lustre-MDD0000: changelog on [10989.954861] Lustre: Skipped 3 previous similar messages [10992.998311] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [10993.909510] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.27051 [10995.411636] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [10996.318486] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.25876 [10997.782077] Lustre: DEBUG MARKER: Test: lfs mkdir -i 1 /mnt/lustre/d205a.sanity.remote [10998.651958] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.13577 [11000.242592] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [11001.175545] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.26440 [11002.722305] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [11003.641069] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.22496 [11005.279512] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [11006.243410] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.27968 [11007.815405] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11008.831183] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.17124 [11011.063735] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [11012.116172] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.16159 [11014.006867] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [11015.127381] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.10576 [11017.011899] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [11018.112107] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.22372 [11020.290215] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [11021.341546] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.2024 [11022.982809] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [11024.000115] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.19176 [11030.188254] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11031.220515] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg356-client.v [11032.727137] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11033.768842] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg356-client.E [11035.386015] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11036.456379] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.oleg356-client.v [11041.734550] Lustre: lustre-MDD0001: changelog off [11041.736761] Lustre: Skipped 3 previous similar messages [11048.006370] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 01:32:48 (1739514768) [11056.099942] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 01:32:56 (1739514776) [11060.297993] Lustre: DEBUG MARKER: == sanity test 205d: verify the format of some stats files ========================================================== 01:33:00 (1739514780) [11070.711849] Lustre: DEBUG MARKER: == sanity test 205e: verify the output of lljobstat ====== 01:33:11 (1739514791) [11084.545249] Lustre: DEBUG MARKER: == sanity test 205f: verify qos_ost_weights YAML format == 01:33:24 (1739514804) [11090.579648] Lustre: DEBUG MARKER: == sanity test 205g: stress test for job_stats procfile == 01:33:30 (1739514810) [11189.310141] Lustre: DEBUG MARKER: == sanity test 205h: check jobid xattr is stored correctly ========================================================== 01:35:09 (1739514909) [11197.823444] Lustre: DEBUG MARKER: == sanity test 205i: check job_xattr parameter accepts and rejects values correctly ========================================================== 01:35:18 (1739514918) [11207.132656] Lustre: DEBUG MARKER: == sanity test 205k: Verify '?' operator on job stats ==== 01:35:27 (1739514927) [11213.857236] Lustre: DEBUG MARKER: == sanity test 205l: Verify job stats can scale ========== 01:35:34 (1739514934) [11294.568414] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 01:36:55 (1739515015) [11298.063269] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 01:36:58 (1739515018) [11301.559383] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 01:37:02 (1739515022) [11305.204554] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 01:37:05 (1739515025) [11312.370051] Lustre: Failing over lustre-MDT0000 [11312.899298] Lustre: server umount lustre-MDT0000 complete [11313.260813] LustreError: 20014:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11313.268091] LustreError: 20014:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 3 previous similar messages [11313.668232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11313.674109] LustreError: Skipped 1 previous similar message [11313.677068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11326.548709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11326.644320] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11326.658849] LustreError: 9847:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88007072b100 x1824009197447552/t0(0) o250->MGC192.168.203.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [11326.837730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11326.873511] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11328.271273] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11328.672744] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [11331.848317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [11331.856944] Lustre: Skipped 1 previous similar message [11331.878842] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11331.932147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:37222 to 0x280000403:37249) [11331.933221] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:37533 to 0x2c0000403:37569) [11333.735564] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11334.473961] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11338.743890] Lustre: Failing over lustre-MDT0000 [11338.901104] Lustre: server umount lustre-MDT0000 complete [11341.876738] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11341.879238] LustreError: 28261:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11341.879240] LustreError: 28261:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 15 previous similar messages [11341.898714] Lustre: Skipped 5 previous similar messages [11352.494083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11352.582519] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11352.740041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11352.767688] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11353.297523] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11354.176924] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [11357.735551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [11357.737631] Lustre: Skipped 3 previous similar messages [11357.757774] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11357.784370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:37222 to 0x280000403:37281) [11357.790946] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:37533 to 0x2c0000403:37601) [11359.517187] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11360.316404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11365.371935] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 01:38:05 (1739515085) [11380.226572] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 01:38:20 (1739515100) [11386.430999] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 01:38:26 (1739515106) [11391.261256] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 01:38:31 (1739515111) [11406.086576] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 01:38:46 (1739515126) [11428.904672] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 01:39:09 (1739515149) [11433.096596] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 01:39:13 (1739515153) [11445.832921] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with embedded hyphen ('-') ========================================================== 01:39:26 (1739515166) [11451.966297] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 01:39:32 (1739515172) [11452.905351] Lustre: DEBUG MARKER: creating a 10 Mb file [11498.132041] Lustre: DEBUG MARKER: starting reads [11499.134845] Lustre: DEBUG MARKER: truncating the file [11500.288606] Lustre: DEBUG MARKER: killing dd [11501.207741] Lustre: DEBUG MARKER: removing the temporary file [11504.840147] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 01:40:25 (1739515225) [11508.906997] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 01:40:29 (1739515229) [11513.105086] Lustre: *** cfs_fail_loc=229, val=1*** [11515.361110] Lustre: *** cfs_fail_loc=229, val=1*** [11515.365919] Lustre: Skipped 2 previous similar messages [11517.128866] Lustre: *** cfs_fail_loc=229, val=0*** [11526.667758] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 01:40:47 (1739515247) [11531.572839] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 01:40:52 (1739515252) [11535.380792] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 01:40:55 (1739515255) [11539.227103] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 01:40:59 (1739515259) [11543.217256] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 01:41:03 (1739515263) [11543.377010] Lustre: lustre-OST0001: Client 229c334c-a683-4021-ab8e-49eda7daef65 (at 192.168.203.56@tcp) reconnecting [11548.174350] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 01:41:08 (1739515268) [11550.839638] LustreError: 13848:0:(tgt_handler.c:2548:tgt_brw_read()) cfs_fail_timeout id 515 sleeping for 3000ms [11553.846139] LustreError: 13848:0:(tgt_handler.c:2548:tgt_brw_read()) cfs_fail_timeout id 515 awake [11558.424390] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 01:41:18 (1739515278) [11566.837591] Lustre: *** cfs_fail_loc=520, val=57344*** [11566.896265] LNet: *** cfs_fail_loc=e000, val=2147483648*** [11571.859771] Lustre: lustre-OST0000: Client 229c334c-a683-4021-ab8e-49eda7daef65 (at 192.168.203.56@tcp) reconnecting [11581.595903] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 01:41:42 (1739515302) [11583.945672] LustreError: 25841:0:(tgt_handler.c:2548:tgt_brw_read()) cfs_fail_timeout id 515 sleeping for 22000ms [11603.981249] Lustre: lustre-OST0000: Client 229c334c-a683-4021-ab8e-49eda7daef65 (at 192.168.203.56@tcp) reconnecting [11605.521154] LustreError: 25841:0:(tgt_handler.c:2548:tgt_brw_read()) cfs_fail_timeout interrupted [11605.524012] LustreError: 25841:0:(ldlm_lib.c:3515:target_bulk_io()) @@@ bulk READ failed: rc = -107 req@ffff88012e393480 x1824009276112256/t0(0) o3->229c334c-a683-4021-ab8e-49eda7daef65@192.168.203.56@tcp:0/0 lens 488/440 e 0 to 0 dl 1739515324 ref 1 fl Interpret:/200/0 rc 0/0 job:'dd.0' uid:0 gid:0 [11605.540298] Lustre: 25841:0:(service.c:2350:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/1s); client may timeout req@ffff88012e393480 x1824009276112256/t0(0) o3->229c334c-a683-4021-ab8e-49eda7daef65@192.168.203.56@tcp:0/0 lens 488/440 e 0 to 0 dl 1739515324 ref 1 fl Complete:/200/0 rc 0/0 job:'dd.0' uid:0 gid:0 [11609.938866] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [11611.005259] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [11611.957347] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 01:42:12 (1739515332) [11615.875010] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 01:42:16 (1739515336) [11618.912015] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 01:42:19 (1739515339) [11623.200068] Lustre: DEBUG MARKER: == sanity test 226d: verify fid2path with -n and -fn option ========================================================== 01:42:23 (1739515343) [11627.383177] Lustre: DEBUG MARKER: == sanity test 226e: Verify path2fid -0 option with newline and space ========================================================== 01:42:27 (1739515347) [11631.261799] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 01:42:31 (1739515351) [11634.952493] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 01:42:35 (1739515355) [11760.684396] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 01:44:41 (1739515481) [11860.915353] Lustre: Failing over lustre-MDT0000 [11861.223567] Lustre: server umount lustre-MDT0000 complete [11861.635817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11861.638637] LustreError: 24162:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11861.638640] LustreError: 24162:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 13 previous similar messages [11861.663490] Lustre: Skipped 3 previous similar messages [11865.199373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11865.272691] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11865.413962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11865.440756] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11866.829212] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [11870.084192] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11870.427788] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [11870.438632] Lustre: Skipped 3 previous similar messages [11870.463219] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11870.488611] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:43780 to 0x2c0000403:43809) [11870.489813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:43495 to 0x280000403:43521) [11893.798856] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 01:46:54 (1739515614) [12104.715417] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 01:50:25 (1739515825) [12107.272311] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 01:50:28 (1739515828) [12110.611752] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 01:50:31 (1739515831) [12144.282483] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 01:51:04 (1739515864) [12145.473028] Lustre: *** cfs_fail_loc=1801, val=0*** [12145.475042] LustreError: 1553:0:(mdt_reint.c:2513:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240001b78:0x567f:0x0]/f1 failed: rc = -5 [12146.256368] LustreError: 1553:0:(mdd_dir.c:4619:mdd_migrate_cmd_check()) lustre-MDD0000: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush migrate_dir' to finish migration: rc = -1 [12146.264377] LustreError: 1553:0:(mdt_reint.c:2513:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200003ab2:0x2fe9:0x0]/migrate_dir failed: rc = -1 [12150.117911] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [12150.926062] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 01:51:11 (1739515871) [12155.428173] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 01:51:15 (1739515875) [12159.921808] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 01:51:20 (1739515880) [12162.938595] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 01:51:23 (1739515883) [12166.067109] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 01:51:26 (1739515886) [12168.958955] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 01:51:29 (1739515889) [12171.998583] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 01:51:32 (1739515892) [12172.658968] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [12173.476426] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 01:51:34 (1739515894) [12228.973930] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 01:52:29 (1739515949) [12236.391502] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 01:52:36 (1739515956) [12240.406079] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 01:52:41 (1739515961) [12245.709676] LustreError: 1928:0:(mdt_reint.c:2513:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200003ab2:0x3488:0x0]/f73 failed: rc = -114 [12245.716838] LustreError: 1928:0:(mdt_reint.c:2513:mdt_reint_migrate()) Skipped 4 previous similar messages [12247.735871] LustreError: 1928:0:(mdt_reint.c:2513:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200003ab2:0x3488:0x0]/d17 failed: rc = -114 [12247.744388] LustreError: 1928:0:(mdt_reint.c:2513:mdt_reint_migrate()) Skipped 91 previous similar messages [12257.743368] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 01:52:58 (1739515978) [12273.329611] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 01:53:13 (1739515993) [12279.213843] LustreError: 2422:0:(mdt_reint.c:2513:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240001b71:0x21:0x0]/f148 failed: rc = -114 [12279.218168] LustreError: 2422:0:(mdt_reint.c:2513:mdt_reint_migrate()) Skipped 8 previous similar messages [12299.795494] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 01:53:40 (1739516020) [12304.312570] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 01:53:44 (1739516024) [12310.437603] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 01:53:51 (1739516031) [12314.887908] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 01:53:55 (1739516035) [12315.514492] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [12316.367716] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 01:53:57 (1739516037) [12317.085887] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [12317.968092] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 01:53:58 (1739516038) [12323.135098] Lustre: DEBUG MARKER: == sanity test 230x: dir migration check space =========== 01:54:03 (1739516043) [12331.404276] LustreError: 13076:0:(mdt_reint.c:2513:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240001b78:0x5c0c:0x0]/sub38 failed: rc = -28 [12331.411385] LustreError: 13076:0:(mdt_reint.c:2513:mdt_reint_migrate()) Skipped 75 previous similar messages [12347.508595] LustreError: 13076:0:(mdt_reint.c:2513:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240001b78:0x5c0c:0x0]/sub68 failed: rc = -114 [12347.523618] LustreError: 13076:0:(mdt_reint.c:2513:mdt_reint_migrate()) Skipped 2 previous similar messages [12362.957937] Lustre: DEBUG MARKER: == sanity test 230y: unlink dir with bad hash type ======= 01:54:43 (1739516083) [12365.734732] Lustre: *** cfs_fail_loc=1802, val=0*** [12374.593167] Lustre: DEBUG MARKER: == sanity test 230z: resume dir migration with bad hash type ========================================================== 01:54:55 (1739516095) [12377.143896] Lustre: *** cfs_fail_loc=1802, val=0*** [12377.146355] Lustre: Skipped 1 previous similar message [12387.803026] LustreError: 13076:0:(mdt_reint.c:2513:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240001b78:0x5c4d:0x0]/d9 failed: rc = -114 [12387.807133] LustreError: 13076:0:(mdt_reint.c:2513:mdt_reint_migrate()) Skipped 7 previous similar messages [12405.508230] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 01:55:26 (1739516126) [12411.062587] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 01:55:31 (1739516131) [12446.538678] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 01:56:07 (1739516167) [12447.128256] Lustre: *** cfs_fail_loc=31c, val=0*** [12450.146087] Lustre: Failing over lustre-OST0000 [12450.420670] Lustre: server umount lustre-OST0000 complete [12451.481946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12451.486843] Lustre: Skipped 1 previous similar message [12451.489320] LustreError: 14545:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12451.490427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12451.501335] LustreError: 14545:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 5 previous similar messages [12454.509856] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [12454.517054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12454.581497] Lustre: 15350:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [12454.687016] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12454.700265] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [12456.403437] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [12456.575823] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [12456.577404] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.156@tcp (at 0@lo) [12456.577405] Lustre: Skipped 3 previous similar messages [12456.967924] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [12461.072870] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 01:56:21 (1739516181) [12461.726852] Lustre: *** cfs_fail_loc=31c, val=0*** [12461.731501] LustreError: 23601:0:(ldlm_request.c:475:ldlm_cli_enqueue_local()) ### delayed lvb init failed (rc -12) ns: filter-lustre-OST0000_UUID lock: ffff880136251200/0x3fd16619735bbb21 lrc: 2/0,0 mode: --/PR res: [0x280000403:0xac31:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 23601 timeout: 0 lvb_type: 0 [12463.610435] Lustre: Failing over lustre-OST0000 [12463.653723] Lustre: server umount lustre-OST0000 complete [12463.974756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [12467.333249] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [12467.344837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12467.438658] Lustre: 16991:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [12467.581093] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12470.153345] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [12474.324903] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 01:56:35 (1739516195) [12477.508201] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 01:56:38 (1739516198) [12480.835678] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 01:56:41 (1739516201) [12484.585267] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 01:56:45 (1739516205) [12490.301939] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 01:56:50 (1739516210) [12494.523112] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 01:56:55 (1739516215) [12497.902902] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 01:56:58 (1739516218) [12549.727116] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 01:57:50 (1739516270) [12551.268169] Lustre: *** cfs_fail_loc=2100, val=0*** [12558.042987] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 01:57:58 (1739516278) [12567.872973] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 01:58:08 (1739516288) [12570.211015] Lustre: *** cfs_fail_loc=713, val=1*** [12570.219261] Lustre: Skipped 1 previous similar message [12577.581817] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 01:58:18 (1739516298) [12632.334797] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 01:59:13 (1739516353) [12657.737767] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 01:59:38 (1739516378) [12658.503134] Lustre: *** cfs_fail_loc=105, val=0*** [12662.824981] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 01:59:43 (1739516383) [12862.708047] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 02:03:03 (1739516583) [12939.797800] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 02:04:20 (1739516660) [12944.573877] Lustre: DEBUG MARKER: == sanity test 245a: check mdc connection flag/data: multiple modify RPCs ========================================================== 02:04:25 (1739516665) [12948.413546] Lustre: DEBUG MARKER: == sanity test 245b: check osp connection flag/data: multiple modify RPCs ========================================================== 02:04:28 (1739516668) [12954.211526] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 02:04:34 (1739516674) [12960.402755] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 02:04:40 (1739516680) [12964.926329] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 02:04:45 (1739516685) [12970.715299] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 02:04:51 (1739516691) [12976.024519] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 02:04:56 (1739516696) [12980.207536] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 02:05:00 (1739516700) [12981.269144] Lustre: enable_remote_subdir_mount is deprecated, it's always enabled. [12991.224304] Lustre: DEBUG MARKER: == sanity test 247g: striped directory submount revalidate ROOT from cache ========================================================== 02:05:11 (1739516711) [12997.664640] Lustre: DEBUG MARKER: == sanity test 247h: remote directory submount revalidate ROOT from cache ========================================================== 02:05:18 (1739516718) [13006.735014] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 02:05:27 (1739516727) [13086.368268] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 02:06:46 (1739516806) [13117.313396] Lustre: DEBUG MARKER: == sanity test 248c: verify whole file read behavior ===== 02:07:17 (1739516837) [13128.971090] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 02:07:29 (1739516849) [13132.522046] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 02:07:33 (1739516853) [13135.929407] Lustre: DEBUG MARKER: == sanity test 251a: Handling short read and write correctly ========================================================== 02:07:36 (1739516856) [13140.237778] Lustre: DEBUG MARKER: == sanity test 251b: short read restore offset correctly ========================================================== 02:07:40 (1739516860) [13148.562604] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 02:07:49 (1739516869) [13153.359203] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 02:07:54 (1739516874) [13188.096660] LustreError: 20014:0:(lod_qos.c:1380:lod_ost_alloc_specific()) can't lstripe objid [0x200004283:0x44:0x0]: have 0 want 1 [13228.344854] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 02:09:08 (1739516948) [13230.230046] Lustre: lustre-MDD0000: changelog on [13230.231616] Lustre: Skipped 1 previous similar message [13236.409765] Lustre: lustre-MDD0001: changelog off [13236.415182] Lustre: Skipped 1 previous similar message [13239.210173] Lustre: DEBUG MARKER: SKIP: sanity test_255a skipping excluded test 255a (base 255) [13240.115752] Lustre: DEBUG MARKER: SKIP: sanity test_255b skipping excluded test 255b (base 255) [13241.082047] Lustre: DEBUG MARKER: SKIP: sanity test_255c skipping excluded test 255c (base 255) [13242.180036] Lustre: DEBUG MARKER: SKIP: sanity test_256 skipping excluded test 256 [13243.051811] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 02:09:23 (1739516963) [13244.151938] Lustre: *** cfs_fail_loc=161, val=0*** [13245.325965] Lustre: Failing over lustre-MDT0001 [13246.498391] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.56@tcp (stopping) [13246.501039] Lustre: Skipped 1 previous similar message [13247.860007] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13247.865742] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13247.876547] Lustre: Skipped 3 previous similar messages [13247.881634] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13249.006074] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13249.008443] Lustre: Skipped 2 previous similar messages [13251.421082] Lustre: server umount lustre-MDT0001 complete [13251.480822] LustreError: 1540:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13251.497185] LustreError: 1540:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 4 previous similar messages [13255.279788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13255.358317] Lustre: 2452:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [13255.582606] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13255.615106] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [13255.625556] Lustre: Skipped 1 previous similar message [13256.486588] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [13256.491622] Lustre: Skipped 1 previous similar message [13257.120380] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13259.219662] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [13260.582433] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [13260.587732] Lustre: Skipped 3 previous similar messages [13260.606904] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [13260.612755] Lustre: Skipped 1 previous similar message [13260.634329] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:34588 to 0x2c0000400:34625) [13260.634340] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:34842 to 0x280000400:34881) [13268.004598] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 02:09:48 (1739516988) [13272.150599] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 02:09:52 (1739516992) [13276.086855] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 02:09:56 (1739516996) [13286.368420] Lustre: *** cfs_fail_loc=2301, val=0*** [13287.805622] Lustre: Failing over lustre-OST0000 [13287.868335] Lustre: server umount lustre-OST0000 complete [13289.793629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13289.799311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13289.808246] Lustre: Skipped 2 previous similar messages [13289.814164] LustreError: 13844:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13289.819424] LustreError: 13844:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 3 previous similar messages [13292.714122] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13292.726666] LDISKFS-fs (dm-2): 1 truncate cleaned up [13292.728497] LDISKFS-fs (dm-2): recovery complete [13292.738956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13292.819548] Lustre: 5551:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [13292.938437] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13292.948782] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13294.099122] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13294.635640] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13294.636058] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.156@tcp (at 0@lo) [13294.636059] Lustre: Skipped 2 previous similar messages [13295.323149] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13302.689510] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 02:10:23 (1739517023) [13305.933879] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 02:10:26 (1739517026) [13313.652353] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 02:10:34 (1739517034) [13317.040506] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 02:10:37 (1739517037) [13320.738535] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 02:10:41 (1739517041) [13324.132236] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 02:10:44 (1739517044) [13328.486096] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 02:10:49 (1739517049) [13331.199581] Lustre: Increasing provided stripe size to a minimum value 64 [13336.693934] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 02:10:57 (1739517057) [13340.270977] Lustre: *** cfs_fail_loc=168, val=40*** [13341.181442] Lustre: *** cfs_fail_loc=168, val=20*** [13341.184415] Lustre: Skipped 2 previous similar messages [13343.197252] Lustre: *** cfs_fail_loc=168, val=15*** [13343.199112] Lustre: Skipped 5 previous similar messages [13350.219944] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 02:11:10 (1739517070) [13355.405611] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 02:11:15 (1739517075) [13359.088785] Lustre: DEBUG MARKER: == sanity test 270j: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 02:11:19 (1739517079) [13363.223864] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 02:11:23 (1739517083) [13366.999991] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 02:11:27 (1739517087) [13370.743149] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 02:11:31 (1739517091) [13374.358805] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 02:11:34 (1739517094) [13455.684315] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 02:12:56 (1739517176) [13459.183966] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 02:12:59 (1739517179) [13463.087769] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 02:13:03 (1739517183) [13466.886168] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 02:13:07 (1739517187) [13470.260230] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 02:13:11 (1739517191) [13475.656407] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 02:13:16 (1739517196) [13481.131527] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 02:13:21 (1739517201) [13486.571248] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 02:13:27 (1739517207) [13492.547875] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 02:13:33 (1739517213) [13498.124441] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 02:13:38 (1739517218) [13502.513046] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 02:13:43 (1739517223) [13503.899781] LustreError: 13035:0:(mdt_io.c:573:mdt_commitrw_write()) cfs_fail_timeout id 16b sleeping for 2000ms [13505.906708] LustreError: 13035:0:(mdt_io.c:573:mdt_commitrw_write()) cfs_fail_timeout id 16b awake [13509.376455] Lustre: DEBUG MARKER: == sanity test 273c: race writeback and object destroy === 02:13:49 (1739517229) [13514.170058] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 02:13:54 (1739517234) [13520.393292] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 02:14:00 (1739517240) [13521.563243] Lustre: Failing over lustre-OST0000 [13521.570630] LustreError: 18295:0:(obd_class.h:1101:obd_statfs()) Device 22 not setup [13521.733917] Lustre: server umount lustre-OST0000 complete [13521.977940] LustreError: 3436:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13521.987710] LustreError: 3436:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 3 previous similar messages [13523.332578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13523.338776] Lustre: Skipped 1 previous similar message [13526.340492] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13526.353714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13526.489052] Lustre: 19502:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [13526.629545] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13526.647802] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13526.978865] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13528.075226] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.156@tcp (at 0@lo) [13528.075591] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13528.088814] Lustre: Skipped 1 previous similar message [13529.127039] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13531.353488] Lustre: Failing over lustre-OST0000 [13531.364696] LustreError: 20819:0:(obd_class.h:1101:obd_statfs()) Device 22 not setup [13531.454957] Lustre: server umount lustre-OST0000 complete [13536.203159] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13536.214075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13536.335268] Lustre: 22009:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [13536.492312] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13539.701787] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13541.883497] Lustre: Failing over lustre-OST0000 [13541.998396] Lustre: server umount lustre-OST0000 complete [13542.926206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13546.937914] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13546.953706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13547.068889] Lustre: 24491:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [13547.220181] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13549.911854] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13551.809279] Lustre: Failing over lustre-OST0000 [13551.812657] LustreError: 25803:0:(obd_class.h:1101:obd_statfs()) Device 22 not setup [13551.909022] Lustre: server umount lustre-OST0000 complete [13556.158950] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13556.166351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13556.367381] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13558.672791] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13560.475724] Lustre: Failing over lustre-OST0000 [13560.558884] Lustre: server umount lustre-OST0000 complete [13561.395342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13561.407645] Lustre: Skipped 7 previous similar messages [13564.762283] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13564.794936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13564.862368] Lustre: 29487:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [13564.867335] Lustre: 29487:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [13564.897473] LustreError: 29521:0:(obd_class.h:1101:obd_statfs()) Device 22 not setup [13565.007326] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13565.011739] Lustre: Skipped 3 previous similar messages [13566.735069] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13566.738168] Lustre: Skipped 3 previous similar messages [13566.886721] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13566.888914] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.156@tcp (at 0@lo) [13566.888916] Lustre: Skipped 7 previous similar messages [13566.898919] Lustre: Skipped 3 previous similar messages [13567.828641] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13569.800291] Lustre: Failing over lustre-OST0000 [13569.921343] Lustre: server umount lustre-OST0000 complete [13574.895516] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13574.915037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13575.234051] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13575.239002] Lustre: Skipped 1 previous similar message [13577.776741] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13579.721852] Lustre: Failing over lustre-OST0000 [13579.725682] LustreError: 768:0:(obd_class.h:1101:obd_statfs()) Device 22 not setup [13579.885028] Lustre: server umount lustre-OST0000 complete [13584.280890] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13584.288702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13587.310236] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13589.519065] Lustre: Failing over lustre-OST0000 [13589.525282] LustreError: 3326:0:(obd_class.h:1101:obd_statfs()) Device 22 not setup [13589.527832] LustreError: 3326:0:(obd_class.h:1101:obd_statfs()) Skipped 2 previous similar messages [13589.744995] Lustre: server umount lustre-OST0000 complete [13591.530015] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13591.533257] LustreError: Skipped 1 previous similar message [13591.538338] LustreError: 13845:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13591.542984] LustreError: 13845:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 26 previous similar messages [13594.248456] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13594.256231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13597.575946] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13600.000411] Lustre: Failing over lustre-OST0000 [13600.204982] Lustre: server umount lustre-OST0000 complete [13601.378204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13601.380705] LustreError: Skipped 1 previous similar message [13604.658384] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13604.669179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13604.787851] Lustre: 6999:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [13604.801568] Lustre: 6999:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 3 previous similar messages [13607.688606] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13609.938079] Lustre: Failing over lustre-OST0000 [13609.940994] LustreError: 8274:0:(obd_class.h:1101:obd_statfs()) Device 22 not setup [13609.940997] LustreError: 8274:0:(obd_class.h:1101:obd_statfs()) Skipped 3 previous similar messages [13609.996917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13610.043739] Lustre: server umount lustre-OST0000 complete [13614.965722] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13614.971879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13615.208076] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13615.213098] Lustre: Skipped 3 previous similar messages [13617.695217] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13619.471730] Lustre: Failing over lustre-OST0000 [13619.576941] Lustre: server umount lustre-OST0000 complete [13624.211895] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13624.218324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13626.748881] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13628.512154] Lustre: Failing over lustre-OST0000 [13628.623327] Lustre: server umount lustre-OST0000 complete [13629.395377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13629.400646] Lustre: Skipped 13 previous similar messages [13633.021590] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13633.033363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13633.297449] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13633.301019] Lustre: Skipped 6 previous similar messages [13634.448315] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13634.455014] Lustre: Skipped 6 previous similar messages [13634.949032] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13634.949381] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.156@tcp (at 0@lo) [13634.949382] Lustre: Skipped 13 previous similar messages [13634.971264] Lustre: Skipped 6 previous similar messages [13635.949083] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13637.978286] Lustre: Failing over lustre-OST0000 [13638.092107] Lustre: server umount lustre-OST0000 complete [13642.310168] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13642.324016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13642.516312] LustreError: 16984:0:(obd_class.h:1101:obd_statfs()) Device 22 not setup [13642.519034] LustreError: 16984:0:(obd_class.h:1101:obd_statfs()) Skipped 4 previous similar messages [13645.529506] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13647.599598] Lustre: Failing over lustre-OST0000 [13647.622141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13647.625634] Lustre: Skipped 2 previous similar messages [13647.703248] Lustre: server umount lustre-OST0000 complete [13652.242842] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13652.254064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13654.804427] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13656.613704] Lustre: Failing over lustre-OST0000 [13656.679361] Lustre: server umount lustre-OST0000 complete [13661.430079] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13661.443731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13664.129269] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13666.080183] Lustre: Failing over lustre-OST0000 [13666.229702] Lustre: server umount lustre-OST0000 complete [13670.479898] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13670.493524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13670.582054] Lustre: 24344:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [13670.586691] Lustre: 24344:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 6 previous similar messages [13672.855703] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13674.636363] Lustre: Failing over lustre-OST0000 [13674.802713] Lustre: server umount lustre-OST0000 complete [13678.980899] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13678.989824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13681.716272] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13683.491690] Lustre: Failing over lustre-OST0000 [13683.581705] Lustre: server umount lustre-OST0000 complete [13687.964386] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13687.970808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13688.166047] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13688.172783] Lustre: Skipped 7 previous similar messages [13690.463376] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13692.117938] Lustre: Failing over lustre-OST0000 [13692.206525] Lustre: server umount lustre-OST0000 complete [13695.826730] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13695.834844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13698.129565] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13699.647382] Lustre: Failing over lustre-OST0000 [13699.706941] Lustre: server umount lustre-OST0000 complete [13703.236795] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13703.242830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13705.669461] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13710.225806] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 02:17:10 (1739517430) [13713.737345] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 02:17:14 (1739517434) [13715.074343] Lustre: Failing over lustre-MDT0000 [13715.080138] Lustre: Failing over lustre-MDT0001 [13715.170058] LustreError: 3960:0:(tgt_mount.c:1739:server_put_super()) cfs_race id 60c sleeping [13717.310278] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.56@tcp (stopping) [13720.171575] LustreError: 3960:0:(tgt_mount.c:1739:server_put_super()) cfs_fail_race id 60c awake: rc=0 [13720.218457] Lustre: server umount lustre-MDT0001 complete [13722.309122] LustreError: 8552:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13722.315676] LustreError: 8552:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 40 previous similar messages [13733.491283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13733.493678] Lustre: Skipped 10 previous similar messages [13745.403434] Lustre: server umount lustre-MDT0000 complete [13749.550887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13749.666446] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13749.752899] LustreError: 4874:0:(tgt_mount.c:1470:server_start_targets()) cfs_fail_race id 60c waking [13751.901010] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13753.378218] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13757.811700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13757.975682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [13757.981222] LustreError: Skipped 1 previous similar message [13758.073684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13759.422349] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13761.500007] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [13763.073312] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [13763.077950] Lustre: Skipped 9 previous similar messages [13772.413455] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [13772.414034] Lustre: lustre-MDT0001: Recovery over after 0:10, of 2 clients 2 recovered and 0 were evicted. [13772.414035] Lustre: Skipped 8 previous similar messages [13772.433967] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:34632 to 0x2c0000400:34657) [13772.439276] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:34897 to 0x280000400:34913) [13772.477968] Lustre: Skipped 17 previous similar messages [13772.515965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:44123 to 0x280000403:44161) [13772.521665] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:44404 to 0x2c0000403:44449) [13780.449058] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 02:18:21 (1739517501) [13781.665662] LustreError: 4880:0:(llog_osd.c:1480:llog_osd_open()) cfs_race id 15e sleeping [13783.562811] Lustre: Failing over lustre-MDT0000 [13783.664196] LustreError: 7021:0:(tgt_mount.c:128:server_deregister_mount()) cfs_fail_race id 15e waking [13783.678986] LustreError: 4880:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e awake: rc=2991 [13784.689236] LustreError: 7021:0:(tgt_mount.c:128:server_deregister_mount()) cfs_fail_race id 15e waking [13784.714674] Lustre: server umount lustre-MDT0000 complete [13787.523079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13787.527569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13787.538367] Lustre: Skipped 21 previous similar messages [13789.492828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13789.604140] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13789.638571] LustreError: 7624:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13790.754292] LustreError: 7625:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13791.763928] LustreError: 7625:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13793.780444] LustreError: 7625:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13793.784921] LustreError: 7625:0:(llog_osd.c:1480:llog_osd_open()) Skipped 1 previous similar message [13797.851669] LustreError: 7625:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13797.855199] LustreError: 7625:0:(llog_osd.c:1480:llog_osd_open()) Skipped 3 previous similar messages [13801.885461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13801.944887] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13801.952731] Lustre: Skipped 10 previous similar messages [13802.896320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:44123 to 0x280000403:44193) [13802.898029] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:44404 to 0x2c0000403:44481) [13803.706520] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug all all [13805.929001] LustreError: 7626:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13805.931561] LustreError: 7626:0:(llog_osd.c:1480:llog_osd_open()) Skipped 4 previous similar messages [13815.162158] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 02:18:55 (1739517535) [13819.540352] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 02:19:00 (1739517540) [13844.348048] Lustre: DEBUG MARKER: == sanity test 300c: chown [14025.837089] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 02:22:26 (1739517746) [14030.582784] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ========================================================== 02:22:31 (1739517751) [14035.081070] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ========================================================== 02:22:35 (1739517755) [14038.728978] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ========================================================== 02:22:39 (1739517759) [14046.310766] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory ========================================================== 02:22:47 (1739517767) [14053.332070] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory ========================================================== 02:22:53 (1739517773) [14090.640655] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ============ 02:23:31 (1739517811) [14094.509275] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ======== 02:23:35 (1739517815) [14098.613158] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ========================================================== 02:23:39 (1739517819) [14102.871385] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS ========================================================== 02:23:43 (1739517823) [14103.466757] Lustre: DEBUG MARKER: SKIP: sanity test_300m Only for single MDT [14104.163037] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA ========================================================== 02:23:44 (1739517824) [14111.787900] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [14112.687758] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================================== 02:23:53 (1739517833) [14113.286765] Lustre: *** cfs_fail_loc=1704, val=0*** [14113.288920] LustreError: 13115:0:(out_handler.c:921:out_tx_end()) lustre-MDT0001-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:425: rc = -524 [14116.442322] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ========================================================== 02:23:57 (1739517837) [14119.513090] Lustre: DEBUG MARKER: == sanity test 300r: test -1 striped directory =========== 02:24:00 (1739517840) [14122.494307] Lustre: DEBUG MARKER: == sanity test 300s: test lfs mkdir -c without -i ======== 02:24:03 (1739517843) [14125.786446] Lustre: DEBUG MARKER: == sanity test 300t: test max_mdt_stripecount ============ 02:24:06 (1739517846) [14131.627083] Lustre: DEBUG MARKER: == sanity test 300ua: basic overstriped dir sanity test == 02:24:12 (1739517852) [14137.001452] Lustre: DEBUG MARKER: == sanity test 300ub: test MDT overstriping interface [14141.832199] Lustre: DEBUG MARKER: == sanity test 300uc: test MDT overstriping as default [14146.161255] Lustre: DEBUG MARKER: == sanity test 300ud: dir split ========================== 02:24:26 (1739517866) [14151.197860] LustreError: 7632:0:(mdt_reint.c:2513:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200005224:0xb7:0x0]/f73 failed: rc = -114 [14151.203982] LustreError: 7632:0:(mdt_reint.c:2513:mdt_reint_migrate()) Skipped 1 previous similar message [14161.148007] LustreError: 5963:0:(mdt_reint.c:2513:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200005224:0xb7:0x0]/d84 failed: rc = -114 [14161.154643] LustreError: 5963:0:(mdt_reint.c:2513:mdt_reint_migrate()) Skipped 105 previous similar messages [14179.936149] LustreError: 7632:0:(mdt_reint.c:2513:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200005224:0xb7:0x0]/f88 failed: rc = -114 [14179.940635] LustreError: 7632:0:(mdt_reint.c:2513:mdt_reint_migrate()) Skipped 271 previous similar messages [14217.585026] LustreError: 7632:0:(mdt_reint.c:2513:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200005224:0xb7:0x0]/d9 failed: rc = -114 [14217.593831] LustreError: 7632:0:(mdt_reint.c:2513:mdt_reint_migrate()) Skipped 675 previous similar messages [14239.535960] Lustre: DEBUG MARKER: == sanity test 300ue: dir merge ========================== 02:26:00 (1739517960) [14295.284330] Lustre: DEBUG MARKER: == sanity test 300uf: migrate with too many local locks == 02:26:55 (1739518015) [14300.945018] Lustre: DEBUG MARKER: == sanity test 300ug: migrate overstriped dirs =========== 02:27:01 (1739518021) [14301.633989] Lustre: 4898:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 3793 > trans_max 3200 [14301.637146] Lustre: 4898:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 6/24/0, destroy: 6/24/0 [14301.641960] Lustre: 4898:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 31/2640/0 [14301.646100] Lustre: 4898:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 0/0/0 [14301.648980] Lustre: 4898:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 47/938/0, delete: 32/50/0 [14301.651726] Lustre: 4898:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 30/30/0, ref_del: 28/28/0 [14301.654472] CPU: 3 PID: 4898 Comm: mdt_io00_002 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1 [14301.660292] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [14301.666453] Call Trace: [14301.669584] [] dump_stack+0x19/0x1b [14301.673025] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [14301.678148] [] top_trans_start+0x351/0xab0 [ptlrpc] [14301.681362] [] lod_trans_start+0x7f/0x2e0 [lod] [14301.683334] [] mdd_trans_start+0x14/0x20 [mdd] [14301.687394] [] mdd_migrate_object+0x141f/0x1dc0 [mdd] [14301.690049] [] mdd_migrate+0x28/0x30 [mdd] [14301.692536] [] mdt_reint_migrate+0x1b63/0x23f0 [mdt] [14301.697838] [] mdt_reint_rec+0x87/0x240 [mdt] [14301.700179] [] mdt_reint_internal+0x74c/0xbc0 [mdt] [14301.703806] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [14301.707149] [] mdt_reint+0x67/0x150 [mdt] [14301.711005] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [14301.713290] [] ptlrpc_server_handle_request+0x281/0xce0 [ptlrpc] [14301.715872] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [14301.717579] [] ? put_prev_entity+0x31/0x400 [14301.719652] [] ? do_raw_spin_unlock+0x49/0x90 [14301.722647] [] ? _raw_spin_unlock_irq+0xe/0x30 [14301.728292] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [14301.730969] [] kthread+0xe4/0xf0 [14301.733231] [] ? kthread_create_on_node+0x140/0x140 [14301.736036] [] ret_from_fork_nospec_begin+0x7/0x21 [14301.740201] [] ? kthread_create_on_node+0x140/0x140 [14306.110959] Lustre: DEBUG MARKER: == sanity test 300uh: overstripe tunable max_stripes_per_mdt ========================================================== 02:27:06 (1739518026) [14310.791929] Lustre: DEBUG MARKER: == sanity test 300ui: overstripe is not supported on one MDT system ========================================================== 02:27:11 (1739518031) [14311.353326] Lustre: DEBUG MARKER: SKIP: sanity test_300ui 1 MDT only [14312.031640] Lustre: DEBUG MARKER: == sanity test 300uj: overstriped dir with -C -N sanity test ========================================================== 02:27:12 (1739518032) [14315.455877] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ============= 02:27:16 (1739518036) [14318.375380] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open ========================================================== 02:27:19 (1739518039) [14321.449846] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ========================================================== 02:27:22 (1739518042) [14322.116195] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [14323.071645] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ========================================================== 02:27:23 (1739518043) [14355.662405] Lustre: DEBUG MARKER: == sanity test 312: make sure ZFS adjusts its block size by write pattern ========================================================== 02:27:56 (1739518076) [14356.574145] Lustre: DEBUG MARKER: SKIP: sanity test_312 the test only applies to zfs [14357.395380] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ========================================================== 02:27:58 (1739518078) [14357.972920] Lustre: *** cfs_fail_loc=720, val=0*** [14357.975890] LustreError: 13849:0:(osd_handler.c:2213:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [14357.988873] LustreError: 13849:0:(ofd_objects.c:1001:ofd_object_punch()) lustre-OST0000: failed to stop transaction: rc = -5 [14361.768157] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ========================================================== 02:28:02 (1739518082) [14362.973423] Lustre: *** cfs_fail_loc=720, val=0*** [14362.979441] LustreError: 3421:0:(osd_handler.c:2213:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [14362.979767] LustreError: 3425:0:(ofd_objects.c:1068:ofd_destroy()) lustre-OST0000 failed to stop transaction: -5 [14362.980555] LustreError: 3419:0:(ofd_dev.c:1769:ofd_destroy_hdl()) lustre-OST0001: error destroying object [0x2c0000403:0xb2f2:0x0]: -5 [14362.994431] LustreError: 3421:0:(osd_handler.c:2213:osd_trans_stop()) Skipped 2 previous similar messages [14365.832758] LustreError: 7635:0:(osp_sync.c:1155:osp_sync_process_committed()) lustre-OST0000-osc-MDT0000: can't cancel record: rc = -2 [14368.844659] LustreError: 7637:0:(osp_sync.c:1155:osp_sync_process_committed()) lustre-OST0001-osc-MDT0000: can't cancel record: rc = -2 [14368.854153] LustreError: 7637:0:(osp_sync.c:1155:osp_sync_process_committed()) Skipped 1 previous similar message [14374.814127] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ============= 02:28:15 (1739518095) [14383.313513] Lustre: DEBUG MARKER: == sanity test 316: lfs migrate of file with large_xattr enabled ========================================================== 02:28:23 (1739518103) [14386.979918] Lustre: DEBUG MARKER: == sanity test 317: Verify blocks get correctly update after truncate ========================================================== 02:28:27 (1739518107) [14390.302918] Lustre: DEBUG MARKER: == sanity test 318: Verify async readahead tunables ====== 02:28:31 (1739518111) [14393.004095] Lustre: DEBUG MARKER: == sanity test 319: lost lease lock on migrate error ===== 02:28:33 (1739518113) [14398.291759] LustreError: 4898:0:(mdt_reint.c:2513:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240002b13:0x4ba:0x0]/xxx failed: rc = -16 [14398.296617] LustreError: 4898:0:(mdt_reint.c:2513:mdt_reint_migrate()) Skipped 364 previous similar messages [14401.508333] Lustre: DEBUG MARKER: == sanity test 350: force NID mismatch path to be exercised ========================================================== 02:28:42 (1739518122) [14402.087427] LNet: *** cfs_fail_loc=e001, val=100*** [14402.089066] LNet: 4888:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-0@lo matched 1824009213597952 with NID mismatch 0@lo accepted (same peer ffff8800b5349c00) [14402.588644] LNet: *** cfs_fail_loc=e001, val=100*** [14402.590358] LNet: Skipped 55 previous similar messages [14402.591388] LNet: 9843:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-192.168.203.56@tcp matched 1824009213605632 with NID mismatch 192.168.203.56@tcp accepted (same peer ffff88013562f000) [14402.597896] LNet: 9843:0:(lib-ptl.c:160:lnet_try_match_md()) Skipped 55 previous similar messages [14583.925452] Lustre: DEBUG MARKER: == sanity test 360: ldiskfs unlink in a separate thread == 02:31:44 (1739518304) [14600.947240] Lustre: DEBUG MARKER: == sanity test 398a: direct IO should cancel lock otherwise lockless ========================================================== 02:32:01 (1739518321) [14604.022002] Lustre: DEBUG MARKER: == sanity test 398b: DIO and buffer IO race ============== 02:32:04 (1739518324) [14694.744945] Lustre: DEBUG MARKER: == sanity test 398c: run fio to test AIO ================= 02:33:35 (1739518415) [14726.428816] Lustre: DEBUG MARKER: == sanity test 398d: run aiocp to verify block size > stripe size ========================================================== 02:34:07 (1739518447) [14740.697179] Lustre: DEBUG MARKER: == sanity test 398e: O_Direct open cleared by fcntl doesn't cause hang ========================================================== 02:34:21 (1739518461) [14744.034777] Lustre: DEBUG MARKER: == sanity test 398f: verify aio handles ll_direct_rw_pages errors correctly ========================================================== 02:34:24 (1739518464) [14751.878068] Lustre: DEBUG MARKER: == sanity test 398g: verify parallel dio async RPC submission ========================================================== 02:34:32 (1739518472) [14753.106463] LustreError: 27857:0:(tgt_handler.c:2694:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14755.115555] LustreError: 27857:0:(tgt_handler.c:2694:tgt_brw_write()) cfs_fail_timeout id 214 awake [14755.515451] LustreError: 27860:0:(tgt_handler.c:2694:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14755.519715] LustreError: 27860:0:(tgt_handler.c:2694:tgt_brw_write()) Skipped 7 previous similar messages [14757.526578] LustreError: 27860:0:(tgt_handler.c:2694:tgt_brw_write()) cfs_fail_timeout id 214 awake [14757.532660] LustreError: 27860:0:(tgt_handler.c:2694:tgt_brw_write()) Skipped 7 previous similar messages [14757.591430] LustreError: 27860:0:(tgt_handler.c:2694:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14759.601634] LustreError: 27860:0:(tgt_handler.c:2694:tgt_brw_write()) cfs_fail_timeout id 214 awake [14759.670346] LustreError: 27860:0:(tgt_handler.c:2694:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14761.680560] LustreError: 27860:0:(tgt_handler.c:2694:tgt_brw_write()) cfs_fail_timeout id 214 awake [14763.757854] LustreError: 27860:0:(tgt_handler.c:2694:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14763.762622] LustreError: 27860:0:(tgt_handler.c:2694:tgt_brw_write()) Skipped 1 previous similar message [14765.766560] LustreError: 27860:0:(tgt_handler.c:2694:tgt_brw_write()) cfs_fail_timeout id 214 awake [14765.776013] LustreError: 27860:0:(tgt_handler.c:2694:tgt_brw_write()) Skipped 1 previous similar message [14772.634524] LustreError: 27863:0:(tgt_handler.c:2694:tgt_brw_write()) cfs_fail_timeout interrupted [14775.178580] Lustre: DEBUG MARKER: == sanity test 398h: verify correctness of read [14787.169977] Lustre: DEBUG MARKER: == sanity test 398i: verify parallel dio handles ll_direct_rw_pages errors correctly ========================================================== 02:35:07 (1739518507) [14791.826818] Lustre: DEBUG MARKER: == sanity test 398j: test parallel dio where stripe size > rpc_size ========================================================== 02:35:12 (1739518512) [14803.914840] Lustre: DEBUG MARKER: == sanity test 398k: test enospc on first stripe ========= 02:35:24 (1739518524) [14815.069952] Lustre: DEBUG MARKER: SKIP: sanity test_398k 7180064 > 600000 skipping out-of-space test on OST0 [14816.020744] Lustre: DEBUG MARKER: == sanity test 398l: test enospc on intermediate stripe/RPC ========================================================== 02:35:36 (1739518536) [14822.521763] Lustre: DEBUG MARKER: SKIP: sanity test_398l 7160152 > 600000 skipping out-of-space test on OST0 [14837.392431] Lustre: DEBUG MARKER: == sanity test 398m: test RPC failures with parallel dio ========================================================== 02:35:57 (1739518557) [14838.052209] Lustre: *** cfs_fail_loc=20e, val=1*** [14839.086244] Lustre: *** cfs_fail_loc=20e, val=1*** [14839.089604] Lustre: Skipped 7 previous similar messages [14841.125448] Lustre: *** cfs_fail_loc=20e, val=1*** [14841.130505] Lustre: Skipped 4 previous similar messages [14844.142031] Lustre: *** cfs_fail_loc=20e, val=1*** [14844.147058] Lustre: Skipped 2 previous similar messages [14848.186257] Lustre: *** cfs_fail_loc=20e, val=1*** [14848.188697] Lustre: Skipped 4 previous similar messages [14859.223852] Lustre: *** cfs_fail_loc=20e, val=1*** [14859.227221] Lustre: Skipped 5 previous similar messages [14883.287254] Lustre: *** cfs_fail_loc=20e, val=1*** [14883.288471] Lustre: Skipped 11 previous similar messages [14917.405963] Lustre: *** cfs_fail_loc=20f, val=1*** [14917.408428] Lustre: Skipped 38 previous similar messages [14988.506516] Lustre: *** cfs_fail_loc=20e, val=2*** [14988.511897] Lustre: Skipped 55 previous similar messages [15069.676935] Lustre: DEBUG MARKER: == sanity test 398n: test append with parallel DIO ======= 02:39:50 (1739518790) [15081.166706] Lustre: DEBUG MARKER: == sanity test 398o: right kms with DIO ================== 02:40:01 (1739518801) [15084.220359] Lustre: DEBUG MARKER: == sanity test 398p: race aio with buffered i/o ========== 02:40:04 (1739518804) [15113.608032] Lustre: DEBUG MARKER: == sanity test 398q: race dio with buffered i/o ========== 02:40:34 (1739518834) [15204.174099] Lustre: DEBUG MARKER: == sanity test 398r: i/o error on file read ============== 02:42:04 (1739518924) [15205.126497] Lustre: *** cfs_fail_loc=20f, val=0*** [15205.130277] Lustre: Skipped 56 previous similar messages [15263.114050] Lustre: DEBUG MARKER: == sanity test 398s: i/o error on mirror file read ======= 02:43:03 (1739518983) [15267.355760] Lustre: DEBUG MARKER: == sanity test 399a: fake write should not be slower than normal write ========================================================== 02:43:07 (1739518987) [15292.585268] Lustre: *** cfs_fail_loc=238, val=0*** [15319.123264] Lustre: DEBUG MARKER: sanity test_399a: @@@@@@ IGNORE (env=kvm): fake write is slower