[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 290127067 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.447569] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.449921] pid_max: default: 32768 minimum: 301 [ 0.451208] Security Framework initialized [ 0.452411] SELinux: Initializing. [ 0.453812] SELinux: Starting in permissive mode [ 0.455295] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.459524] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.462208] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.463969] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.466550] Initializing cgroup subsys memory [ 0.467835] Initializing cgroup subsys devices [ 0.469309] Initializing cgroup subsys freezer [ 0.470735] Initializing cgroup subsys net_cls [ 0.472088] Initializing cgroup subsys blkio [ 0.473522] Initializing cgroup subsys perf_event [ 0.475031] Initializing cgroup subsys hugetlb [ 0.476626] Initializing cgroup subsys pids [ 0.478139] Initializing cgroup subsys net_prio [ 0.479694] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.482516] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.484484] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.486163] tlb_flushall_shift: 6 [ 0.487868] FEATURE SPEC_CTRL Present [ 0.488824] FEATURE IBPB_SUPPORT Present [ 0.490247] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.492464] Spectre V2 : Vulnerable [ 0.493707] Speculative Store Bypass: Vulnerable [ 0.496326] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.504676] ACPI: Core revision 20130517 [ 0.507555] ACPI: All ACPI Tables successfully acquired [ 0.509265] ftrace: allocating 30294 entries in 119 pages [ 0.563815] Enabling x2apic [ 0.564794] Enabled x2apic [ 0.566209] Switched APIC routing to physical x2apic. [ 0.571277] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.573494] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.578081] TSC deadline timer enabled [ 0.578257] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.580983] ... version: 2 [ 0.582230] ... bit width: 48 [ 0.583644] ... generic registers: 4 [ 0.584880] ... value mask: 0000ffffffffffff [ 0.586553] ... max period: 00007fffffffffff [ 0.588200] ... fixed-purpose events: 3 [ 0.589469] ... event mask: 000000070000000f [ 0.591305] KVM setup paravirtual spinlock [ 0.596652] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.600493] KVM setup async PF for cpu 1 [ 0.602062] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.605257] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.608936] KVM setup async PF for cpu 2 [ 0.594791] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.609935] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.614359] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.616021] Brought up 4 CPUs [ 0.616051] KVM setup async PF for cpu 3 [ 0.616061] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.619199] smpboot: Max logical packages: 1 [ 0.620306] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.624831] devtmpfs: initialized [ 0.626176] x86/mm: Memory block size: 128MB [ 0.630424] EVM: security.selinux [ 0.631236] EVM: security.ima [ 0.631848] EVM: security.capability [ 0.635069] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.637607] NET: Registered protocol family 16 [ 0.639120] cpuidle: using governor haltpoll [ 0.640662] ACPI: bus type PCI registered [ 0.641706] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.643562] PCI: Using configuration type 1 for base access [ 0.645372] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.654558] ACPI: Added _OSI(Module Device) [ 0.655934] ACPI: Added _OSI(Processor Device) [ 0.657398] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.658610] ACPI: Added _OSI(Processor Aggregator Device) [ 0.659839] ACPI: Added _OSI(Linux-Dell-Video) [ 0.661711] ACPI: EC: Look up EC in DSDT [ 0.664009] ACPI: Interpreter enabled [ 0.665029] ACPI: (supports S0 S3 S4 S5) [ 0.665811] ACPI: Using IOAPIC for interrupt routing [ 0.666766] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.669462] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.676279] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.678339] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.680681] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.683032] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.687076] acpiphp: Slot [2] registered [ 0.688461] acpiphp: Slot [3] registered [ 0.689697] acpiphp: Slot [4] registered [ 0.691209] acpiphp: Slot [5] registered [ 0.692557] acpiphp: Slot [6] registered [ 0.693953] acpiphp: Slot [7] registered [ 0.695348] acpiphp: Slot [8] registered [ 0.696718] acpiphp: Slot [9] registered [ 0.697896] acpiphp: Slot [10] registered [ 0.699273] acpiphp: Slot [11] registered [ 0.700742] acpiphp: Slot [12] registered [ 0.701976] acpiphp: Slot [13] registered [ 0.703514] acpiphp: Slot [14] registered [ 0.705063] acpiphp: Slot [15] registered [ 0.706595] acpiphp: Slot [16] registered [ 0.708122] acpiphp: Slot [17] registered [ 0.709461] acpiphp: Slot [18] registered [ 0.711012] acpiphp: Slot [19] registered [ 0.712629] acpiphp: Slot [20] registered [ 0.713491] acpiphp: Slot [21] registered [ 0.714926] acpiphp: Slot [22] registered [ 0.716455] acpiphp: Slot [23] registered [ 0.717984] acpiphp: Slot [24] registered [ 0.719509] acpiphp: Slot [25] registered [ 0.721085] acpiphp: Slot [26] registered [ 0.722569] acpiphp: Slot [27] registered [ 0.724152] acpiphp: Slot [28] registered [ 0.725609] acpiphp: Slot [29] registered [ 0.726781] acpiphp: Slot [30] registered [ 0.728139] acpiphp: Slot [31] registered [ 0.729405] PCI host bridge to bus 0000:00 [ 0.730735] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.732878] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.735021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.737327] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.739889] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.742380] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.744227] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.745066] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.746089] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.752183] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.755567] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.757922] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.759977] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.762306] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.765018] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.765707] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.767997] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.770603] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.772439] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.782361] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.784118] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.785253] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.794067] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.798382] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.813555] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.820511] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.823412] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.826333] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.838742] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.844332] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.848887] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.854189] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.861998] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.870800] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.875519] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.877957] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.886405] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.890083] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.892717] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.897967] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.916169] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.920706] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.923936] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.927267] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.937933] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.946285] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.948797] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.950873] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.952882] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.954941] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.957428] vgaarb: loaded [ 0.958626] SCSI subsystem initialized [ 0.960020] ACPI: bus type USB registered [ 0.961506] usbcore: registered new interface driver usbfs [ 0.963078] usbcore: registered new interface driver hub [ 0.964855] usbcore: registered new device driver usb [ 0.968337] PCI: Using ACPI for IRQ routing [ 0.969847] PCI: pci_cache_line_size set to 64 bytes [ 0.970178] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.970185] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 0.970187] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 0.970888] NetLabel: Initializing [ 0.972046] NetLabel: domain hash size = 128 [ 0.973444] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.975133] NetLabel: unlabeled traffic allowed by default [ 0.978845] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.980671] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.985714] amd_nb: Cannot enumerate AMD northbridges [ 0.987669] Switched to clocksource kvm-clock [ 1.005163] pnp: PnP ACPI init [ 1.006270] ACPI: bus type PNP registered [ 1.007639] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.007726] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.007767] pnp 00:02: [dma 2] [ 1.007803] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.007865] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.007928] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.007992] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.008324] pnp: PnP ACPI: found 6 devices [ 1.009434] ACPI: bus type PNP unregistered [ 1.020740] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.020748] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.020751] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.020754] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.020756] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.020840] NET: Registered protocol family 2 [ 1.022943] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.025126] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.027881] TCP: Hash tables configured (established 32768 bind 32768) [ 1.029690] TCP: reno registered [ 1.030603] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.032294] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.034378] NET: Registered protocol family 1 [ 1.036223] RPC: Registered named UNIX socket transport module. [ 1.038288] RPC: Registered udp transport module. [ 1.039920] RPC: Registered tcp transport module. [ 1.041256] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.043400] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.045385] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.046631] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.048599] PCI: CLS 0 bytes, default 64 [ 1.048774] Unpacking initramfs... [ 2.407709] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.411293] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.413460] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.416942] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.420103] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.422019] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.424016] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.428305] cryptomgr_test (50) used greatest stack depth: 14480 bytes left [ 2.431804] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.433636] Initialise system trusted keyring [ 2.465808] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.468111] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.474749] zpool: loaded [ 2.475819] zbud: loaded [ 2.477198] VFS: Disk quotas dquot_6.6.0 [ 2.478759] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.481803] NFS: Registering the id_resolver key type [ 2.483417] Key type id_resolver registered [ 2.484709] Key type id_legacy registered [ 2.485927] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.488423] Key type big_key registered [ 2.489696] SELinux: Registering netfilter hooks [ 2.493703] cryptomgr_test (56) used greatest stack depth: 14048 bytes left [ 2.496346] cryptomgr_test (57) used greatest stack depth: 13968 bytes left [ 2.498987] cryptomgr_test (59) used greatest stack depth: 13664 bytes left [ 2.499556] NET: Registered protocol family 38 [ 2.499576] Key type asymmetric registered [ 2.499580] Asymmetric key parser 'x509' registered [ 2.499718] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.499841] io scheduler noop registered [ 2.499846] io scheduler deadline registered (default) [ 2.499944] io scheduler cfq registered [ 2.499949] io scheduler mq-deadline registered [ 2.499953] io scheduler kyber registered [ 2.502271] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.502289] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.519468] intel_idle: does not run on family 6 model 62 [ 2.519611] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.521962] ACPI: Power Button [PWRF] [ 2.523699] GHES: HEST is not enabled! [ 2.586511] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.648661] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.773608] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.838644] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 2.963434] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.996503] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 3.033011] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 3.037087] Non-volatile memory driver v1.3 [ 3.038738] Linux agpgart interface v0.103 [ 3.040593] crash memory driver: version 1.1 [ 3.042706] nbd: registered device at major 43 [ 3.049387] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 3.049437] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 3.049477] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 3.049516] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 3.049555] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 3.056921] virtio_blk virtio1: [vda] 115440 512-byte logical blocks (59.1 MB/56.3 MiB) [ 3.064414] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 3.064464] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 3.064510] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 3.064551] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 3.064619] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 3.073231] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.081981] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 3.082026] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 3.082066] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 3.082110] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 3.082156] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 3.088003] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.095750] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 3.095804] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 3.095849] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 3.095898] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 3.095941] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 3.104023] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.110811] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 3.110861] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 3.110915] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 3.110961] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 3.111070] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 3.117250] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.123641] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 3.123708] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 3.123765] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 3.123812] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 3.123853] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 3.130384] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.137615] rdac: device handler registered [ 3.139296] hp_sw: device handler registered [ 3.140725] emc: device handler registered [ 3.142082] libphy: Fixed MDIO Bus: probed [ 3.149144] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.151363] ehci-pci: EHCI PCI platform driver [ 3.152976] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.155080] ohci-pci: OHCI PCI platform driver [ 3.156590] uhci_hcd: USB Universal Host Controller Interface driver [ 3.158767] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.161539] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.162798] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.165064] mousedev: PS/2 mouse device common for all mice [ 3.167500] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.171112] rtc_cmos 00:05: RTC can wake from S4 [ 3.174525] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.177741] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.182140] hidraw: raw HID events driver (C) Jiri Kosina [ 3.184371] usbcore: registered new interface driver usbhid [ 3.186211] usbhid: USB HID core driver [ 3.187566] drop_monitor: Initializing network drop monitor service [ 3.189224] Netfilter messages via NETLINK v0.30. [ 3.190298] TCP: cubic registered [ 3.191495] Initializing XFRM netlink socket [ 3.193181] NET: Registered protocol family 10 [ 3.196313] NET: Registered protocol family 17 [ 3.197984] Key type dns_resolver registered [ 3.200214] mce: Using 10 MCE banks [ 3.202303] Loading compiled-in X.509 certificates [ 3.205067] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.208647] registered taskstats version 1 [ 3.213469] modprobe (71) used greatest stack depth: 13456 bytes left [ 3.220069] Key type trusted registered [ 3.228155] Key type encrypted registered [ 3.229865] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.234344] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.238799] rtc_cmos 00:05: setting system clock to 2025-05-01 20:25:56 UTC (1746131156) [ 3.242497] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.245234] Write protecting the kernel read-only data: 12288k [ 3.247529] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.250112] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.259982] random: systemd: uninitialized urandom read (16 bytes read) [ 3.264238] random: systemd: uninitialized urandom read (16 bytes read) [ 3.266556] random: systemd: uninitialized urandom read (16 bytes read) [ 3.270987] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.277443] systemd[1]: Detected virtualization kvm. [ 3.279394] systemd[1]: Detected architecture x86-64. [ 3.281303] systemd[1]: Running in initial RAM disk. [ 3.286170] systemd[1]: No hostname configured. [ 3.287575] systemd[1]: Set hostname to . [ 3.289489] random: systemd: uninitialized urandom read (16 bytes read) [ 3.292814] systemd[1]: Initializing machine ID from random generator. [ 3.307271] ln (87) used greatest stack depth: 13008 bytes left [ 3.348495] random: systemd: uninitialized urandom read (16 bytes read) [ 3.350889] random: systemd: uninitialized urandom read (16 bytes read) [ 3.353213] random: systemd: uninitialized urandom read (16 bytes read) [ 3.355222] random: systemd: uninitialized urandom read (16 bytes read) [ 3.359222] random: systemd: uninitialized urandom read (16 bytes read) [ 3.361864] random: systemd: uninitialized urandom read (16 bytes read) [ 3.372255] systemd[1]: Created slice Root Slice. [ 3.376268] systemd[1]: Created slice System Slice. [ 3.381429] systemd[1]: Reached target Slices. [ 3.385646] systemd[1]: Listening on Journal Socket. [ 3.391385] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.400881] systemd[1]: Starting Load Kernel Modules... [ 3.406035] systemd[1]: Listening on udev Kernel Socket. [ 3.412038] systemd[1]: Starting dracut cmdline hook... [ 3.418193] systemd[1]: Starting Setup Virtual Console... [ 3.422749] systemd[1]: Reached target Local File Systems. [ 3.427797] systemd[1]: Reached target Swap. [ 3.429459] tsc: Refined TSC clocksource calibration: 2399.975 MHz [ 3.435214] systemd[1]: Reached target Timers. [ 3.440641] systemd[1]: Starting Journal Service... [ 3.445224] systemd[1]: Listening on udev Control Socket. [ 3.449788] systemd[1]: Reached target Sockets. [ 3.455364] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 3.466934] systemd[1]: Started Load Kernel Modules. [ 3.472316] systemd[1]: Started Setup Virtual Console. [ 3.478227] systemd[1]: Started Journal Service. [ 3.880712] random: fast init done [ 4.024128] libata version 3.00 loaded. [ 4.026984] ata_piix 0000:00:01.1: version 2.13 [ 4.028907] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 4.041471] scsi host0: ata_piix [ 4.054508] scsi host1: ata_piix [ 4.055645] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.059678] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 4.161378] ip (343) used greatest stack depth: 12336 bytes left [ 7.253148] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 7.673103] systemd-journald[112]: Received SIGTERM from PID 1 (systemd). [ 7.938809] SELinux: Disabled at runtime. [ 7.940579] SELinux: Unregistering netfilter hooks [ 8.010542] ip_tables: (C) 2000-2006 Netfilter Core Team [ 8.014244] systemd[1]: Inserted module 'ip_tables' [ 8.469357] systemd-journald[564]: Received request to flush runtime journal from PID 1 [ 8.614186] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 8.621244] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 8.675132] cryptd: max_cpu_qlen set to 1000 [ 8.714196] AVX version of gcm_enc/dec engaged. [ 8.716812] AES CTR mode by8 optimization enabled [ 8.725189] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 8.761939] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 8.771168] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 8.781567] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 8.910834] EDAC MC: Ver: 3.0.0 [ 8.920243] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 8.920257] EDAC sbridge: Ver: 1.1.2 [ 8.941952] systemctl (720) used greatest stack depth: 11824 bytes left [ 10.864597] mount.nfs (770) used greatest stack depth: 10704 bytes left [ 22.166127] device-mapper: uevent: version 1.0.3 [ 22.168098] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 27.095358] libcfs: loading out-of-tree module taints kernel. [ 27.097862] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 27.131270] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_hostid [ 33.813840] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 34.239951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 34.244841] alg: No test for adler32 (adler32-zlib) [ 35.288756] Lustre: Lustre: Build Version: 2.16.54_84_g4581c23 [ 35.605409] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 35.607687] LNet: Accept secure, port 988 [ 37.234831] Key type lgssc registered [ 37.867308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 43.373746] icp: module license 'CDDL' taints kernel. [ 43.376131] Disabling lock debugging due to kernel taint [ 45.587947] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 49.842943] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 57.391738] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 60.930944] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 60.939918] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 64.888932] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 64.896832] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 71.236153] hrtimer: interrupt took 9481518 ns [ 71.652519] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 78.107911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 78.131139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 78.140645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 79.292798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 79.326218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 79.365128] Lustre: lustre-MDT0000: new disk, initializing [ 79.424514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 79.432305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 79.511639] mount.lustre (7216) used greatest stack depth: 10208 bytes left [ 81.164077] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 88.043259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 88.087463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 88.140078] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 88.150506] Lustre: 8375:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 88.162659] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 88.164274] Lustre: Skipped 1 previous similar message [ 88.214133] Lustre: lustre-MDT0001: new disk, initializing [ 88.254371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 88.268290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 88.276579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 89.933775] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 93.025369] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 97.786397] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 97.802164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 97.843423] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 97.853434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 98.014116] Lustre: 9577:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 98.028318] Lustre: lustre-OST0000: new disk, initializing [ 98.030837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 98.058935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 100.431435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 100.435067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 100.514748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 100.966495] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 108.999719] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 109.008229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 109.035571] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 109.043919] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 109.090419] Lustre: 10920:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 109.106391] Lustre: lustre-OST0001: new disk, initializing [ 109.108725] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 109.134191] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 111.431892] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 117.321995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 117.327605] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 117.379268] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 118.638134] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 125.342567] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 130.646337] random: crng init done [ 133.015470] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing check_logdir /tmp/testlogs/ [ 135.178043] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing yml_node [ 137.753329] Lustre: DEBUG MARKER: Client: 2.16.54.84 [ 139.027538] Lustre: DEBUG MARKER: MDS: 2.16.54.84 [ 140.251250] Lustre: DEBUG MARKER: OSS: 2.16.54.84 [ 141.041938] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Thu May 1 16:28:13 EDT 2025 [ 150.950459] Lustre: DEBUG MARKER: - need mds1 <= 2.14.55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 151.762140] Lustre: DEBUG MARKER: - need mds1 < v2_14_55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 152.505924] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42c 42b 118c 118d 407 119i 851 411b [ 153.405237] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 [ 154.257847] Lustre: DEBUG MARKER: === sanity: start setup 16:28:26 (1746131306) === [ 156.112784] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing check_config_client /mnt/lustre [ 165.971216] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 167.967465] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 169.631198] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.556568] Lustre: DEBUG MARKER: === sanity: finish setup 16:28:44 (1746131324) === [ 176.611879] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 16:28:48 (1746131328) [ 178.285168] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 179.207371] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 16:28:51 (1746131331) [ 182.543578] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 16:28:54 (1746131334) [ 281.130598] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 16:30:33 (1746131433) [ 283.534890] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 16:30:35 (1746131435) [ 284.001737] Lustre: *** cfs_fail_loc=15b, val=0*** [ 284.003914] Lustre: *** cfs_fail_loc=15b, val=0*** [ 286.087542] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 16:30:38 (1746131438) [ 288.948532] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 16:30:41 (1746131441) [ 289.366634] Lustre: *** cfs_fail_loc=19a, val=0*** [ 290.143174] Lustre: *** cfs_fail_loc=19a, val=0*** [ 290.144827] Lustre: Skipped 1 previous similar message [ 291.264481] Lustre: *** cfs_fail_loc=19a, val=0*** [ 291.266522] Lustre: Skipped 2 previous similar messages [ 293.383871] Lustre: *** cfs_fail_loc=19a, val=0*** [ 293.385943] Lustre: Skipped 5 previous similar messages [ 297.458607] Lustre: *** cfs_fail_loc=19a, val=0*** [ 297.460560] Lustre: Skipped 12 previous similar messages [ 305.691181] Lustre: *** cfs_fail_loc=19a, val=0*** [ 305.692925] Lustre: Skipped 22 previous similar messages [ 321.830875] Lustre: *** cfs_fail_loc=19a, val=0*** [ 321.833782] Lustre: Skipped 45 previous similar messages [ 332.251858] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 16:31:24 (1746131484) [ 332.707015] Lustre: *** cfs_fail_loc=188, val=0*** [ 334.208395] Lustre: *** cfs_fail_loc=189, val=0*** [ 338.077228] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 338.890957] Lustre: DEBUG MARKER: == sanity test 60j: llog_reader reports corruptions ====== 16:31:31 (1746131491) [ 341.531411] Lustre: lustre-MDD0000: changelog on [ 342.790947] Lustre: lustre-MDD0001: changelog on [ 347.335868] Lustre: DEBUG MARKER: SKIP: sanity test_60j path oi.1/0x1:0xb:0x0 is not in 'O/1/d/' format [ 348.612472] Lustre: lustre-MDD0001: changelog off [ 350.057763] Lustre: lustre-MDD0000: changelog off [ 351.564146] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 16:31:43 (1746131503) [ 354.757603] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 16:31:46 (1746131506) [ 358.127085] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 16:31:50 (1746131510) [ 422.084108] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 16:32:54 (1746131574) [ 428.981605] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 16:33:01 (1746131581) [ 433.034778] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 434.009022] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 16:33:06 (1746131586) [ 438.557377] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 16:33:10 (1746131590) [ 462.378194] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 16:33:34 (1746131614) [ 464.307800] Lustre: *** cfs_fail_loc=725, val=0*** [ 470.357042] Lustre: *** cfs_fail_loc=725, val=0*** [ 475.205836] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 16:33:47 (1746131627) [ 481.230415] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 16:33:53 (1746131633) [ 487.912153] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 16:33:59 (1746131639) [ 502.308294] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 16:34:14 (1746131654) [ 506.816282] Lustre: *** cfs_fail_loc=513, val=17*** [ 506.822399] LustreError: 17275:0:(service.c:2266:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1830951255570048 [ 507.887154] Lustre: Failing over lustre-OST0000 [ 507.923614] LustreError: 31876:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 508.010459] Lustre: server umount lustre-OST0000 complete [ 508.025504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.035770] Lustre: Skipped 1 previous similar message [ 513.029555] LustreError: 17274:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.052995] LustreError: 17274:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 517.193536] LustreError: 17274:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.118247] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 522.131617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 522.190720] Lustre: 32415:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 522.207096] LustreError: 17274:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.214590] LustreError: 17274:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 522.271520] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 522.300779] mount.lustre (32415) used greatest stack depth: 9872 bytes left [ 523.532854] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 524.833615] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 524.842761] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.147@tcp (at 0@lo) [ 525.152751] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [ 529.180004] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 529.799978] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 534.952148] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 16:34:47 (1746131687) [ 538.362400] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 16:34:50 (1746131690) [ 541.656483] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 16:34:53 (1746131693) [ 544.630641] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 16:34:56 (1746131696) [ 547.772472] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 16:34:59 (1746131699) [ 551.253525] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 16:35:03 (1746131703) [ 554.642937] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 16:35:06 (1746131706) [ 558.354697] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 16:35:10 (1746131710) [ 561.898508] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 16:35:14 (1746131714) [ 566.183223] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 16:35:18 (1746131718) [ 570.434169] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 16:35:22 (1746131722) [ 571.611016] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.618114] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 571.624288] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [ 571.635368] Lustre: Skipped 1 previous similar message [ 572.119258] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 573.043273] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.048580] Lustre: Skipped 2 previous similar messages [ 573.053250] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.202.147@tcp (at 0@lo) [ 573.057060] Lustre: Skipped 2 previous similar messages [ 583.310450] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 593.692944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.722442] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 593.734851] Lustre: Skipped 2 previous similar messages [ 593.742626] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 593.750543] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [ 595.827943] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 595.976740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 597.660951] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 597.768499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 607.622843] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 615.285825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.291486] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 615.295702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 615.302260] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.147@tcp (at 0@lo) [ 616.396451] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 616.465061] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 617.811461] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 617.943956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 626.689639] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 635.089253] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.096771] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 635.102837] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 635.110472] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [ 636.641209] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 636.757958] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 638.109997] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 638.183540] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 646.081263] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 654.230097] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.244469] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 654.251071] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 654.259523] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.202.147@tcp (at 0@lo) [ 655.961796] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 656.059818] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 657.519277] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 657.601500] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 660.074119] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 16:36:52 (1746131812) [ 662.695450] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 16:36:55 (1746131815) [ 664.626959] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 16:36:57 (1746131817) [ 681.602559] Lustre: DEBUG MARKER: == sanity test 65o: pool inheritance for mdt component === 16:37:13 (1746131833) [ 697.607224] Lustre: DEBUG MARKER: == sanity test 65p: setstripe with yaml file and huge number ========================================================== 16:37:29 (1746131849) [ 699.943061] Lustre: DEBUG MARKER: == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 16:37:32 (1746131852) [ 702.051281] Lustre: DEBUG MARKER: == sanity test 65r: prevent all-zero offsets ============= 16:37:34 (1746131854) [ 704.381406] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 16:37:36 (1746131856) [ 708.557997] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 16:37:40 (1746131860) [ 709.152429] Lustre: *** cfs_fail_loc=217, val=0*** [ 709.156160] Lustre: Skipped 1 previous similar message [ 710.004829] Lustre: *** cfs_fail_loc=217, val=0*** [ 712.590099] Lustre: DEBUG MARKER: == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 16:37:44 (1746131864) [ 717.440936] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 717.925621] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 16:37:50 (1746131870) [ 720.686077] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 16:37:52 (1746131872) [ 724.638148] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 16:37:56 (1746131876) [ 754.125641] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 16:38:26 (1746131906) [ 756.994699] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 16:38:29 (1746131909) [ 759.823226] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 16:38:32 (1746131912) [ 762.845454] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 16:38:35 (1746131915) [ 805.211766] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 16:39:17 (1746131957) [ 834.235858] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 16:39:46 (1746131986) [ 838.304231] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 16:39:50 (1746131990) [ 838.694285] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.47@tcp inode [0x200000407:0xc1a:0x0] object 0x280000401:3836 extent [0-4194303]: client csum 65557fec, server csum 65557feb [ 840.505297] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 841.745127] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.47@tcp inode [0x200000407:0xc1a:0x0] object 0x280000401:3836 extent [0-4194303], client returned csum 36e2e89f (type 1), server csum 5a174063 (type 1) [ 842.935505] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 844.152648] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.47@tcp inode [0x200000407:0xc1a:0x0] object 0x280000401:3836 extent [0-4194303], client returned csum f5e48b9b (type 2), server csum 9f2f8bc8 (type 2) [ 845.116285] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 846.313226] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.47@tcp inode [0x200000407:0xc1a:0x0] object 0x280000401:3836 extent [0-4194303], client returned csum b41a0885 (type 4), server csum 65557feb (type 4) [ 847.227472] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 848.398629] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.47@tcp inode [0x200000407:0xc1a:0x0] object 0x280000401:3836 extent [0-4194303], client returned csum ea59fa04 (type 10), server csum a97af9d7 (type 10) [ 849.544340] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 851.930212] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 853.175527] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.47@tcp inode [0x200000407:0xc1a:0x0] object 0x280000401:3836 extent [0-4194303], client returned csum f6c6af15 (type 40), server csum 7445ae4b (type 40) [ 853.186956] LustreError: Skipped 1 previous similar message [ 854.378271] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 856.680761] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 859.600312] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 16:40:11 (1746132011) [ 862.489522] Lustre: 9589:0:(tgt_handler.c:1935:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000407:0xc1b:0x0]:[0-1048575]-2f91fff2-4b7ac6a9 [ 862.504257] LustreError: dumping log to /tmp/lustre-log.1746132015.9589 [ 862.566549] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.202.47@tcp inode [0x200000407:0xc1b:0x0] object 0x280000401:3837 extent [0-1048575], client returned csum 2f91fff2 (type 4), server csum 4b7ac6a9 (type 4) [ 862.574685] LustreError: Skipped 1 previous similar message [ 872.560764] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 16:40:24 (1746132024) [ 872.753138] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.202.47@tcp inode [0x200000407:0xc1d:0x0] object 0x2c0000401:3828 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 878.301063] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 16:40:30 (1746132030) [ 879.022990] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 879.113615] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.47@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 882.296770] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.47@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 882.308495] LustreError: Skipped 3 previous similar messages [ 889.474819] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.47@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 889.481920] LustreError: Skipped 2 previous similar messages [ 900.888985] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.47@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 900.911365] LustreError: Skipped 2 previous similar messages [ 924.915116] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.47@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 924.922142] LustreError: Skipped 5 previous similar messages [ 935.544579] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 957.009128] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.47@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [4194304-8388607]: client csum 7d33b9a0, server csum 7d33b99f [ 957.015570] LustreError: Skipped 15 previous similar messages [ 991.806822] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1028.059549] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.47@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [4194304-8388607]: client csum de2bf73f, server csum de2bf73e [ 1028.071747] LustreError: Skipped 25 previous similar messages [ 1047.911448] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1104.059454] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1159.702101] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.47@tcp inode [0x200000407:0xc1e:0x0] object 0x280000401:3838 extent [4194304-8388607]: client csum fec05011, server csum fec05010 [ 1159.718939] LustreError: Skipped 46 previous similar messages [ 1160.588355] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1216.146170] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1272.265793] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1273.917603] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 16:47:06 (1746132426) [ 1274.344638] Lustre: *** cfs_fail_loc=21a, val=0*** [ 1276.345590] Lustre: *** cfs_fail_loc=21b, val=0*** [ 1279.941464] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 16:47:12 (1746132432) [ 1280.261004] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1282.895935] Lustre: Disabling parameter lustre.osc.lustre*.checksums in log params [ 1282.898947] Lustre: Skipped 1 previous similar message [ 1286.038311] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1288.527298] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 16:47:20 (1746132440) [ 1288.964565] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 1289.461545] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1290.618522] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid 50 [ 1295.285924] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid in IDLE state after 4 sec [ 1296.399814] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid 50 [ 1296.959219] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid in FULL state after 0 sec [ 1297.617027] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1298.981641] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid 50 [ 1310.615365] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid in IDLE state after 11 sec [ 1311.532103] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid 50 [ 1311.945694] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid in FULL state after 0 sec [ 1312.463630] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1313.555779] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid 50 [ 1325.136012] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid in IDLE state after 11 sec [ 1326.107968] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid 50 [ 1326.552380] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid in FULL state after 0 sec [ 1326.961009] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1327.939722] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid 50 [ 1340.720293] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid in IDLE state after 12 sec [ 1341.942497] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid 50 [ 1342.474783] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid in FULL state after 0 sec [ 1343.026783] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1344.279280] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid 50 [ 1355.127913] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid in IDLE state after 10 sec [ 1356.507031] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid 50 [ 1356.957699] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid in FULL state after 0 sec [ 1357.417536] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1358.579928] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid 50 [ 1370.311477] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid in IDLE state after 11 sec [ 1371.459221] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid 50 [ 1371.992163] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid in FULL state after 0 sec [ 1372.716422] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1374.094436] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid 50 [ 1385.750230] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid in IDLE state after 11 sec [ 1386.659327] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid 50 [ 1387.070073] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b6166800.ost_server_uuid in FULL state after 0 sec [ 1389.070104] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1389.605069] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 16:49:01 (1746132541) [ 1391.555396] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 16:49:03 (1746132543) [ 1392.167190] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1392.676970] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1393.180358] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1393.635804] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1395.677247] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1396.259184] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 16:49:08 (1746132548) [ 1399.835089] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 16:49:12 (1746132552) [ 1402.184056] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check =========== 16:49:14 (1746132554) [ 1415.644741] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 16:49:28 (1746132568) [ 1418.420002] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 16:49:30 (1746132570) [ 1418.793786] Lustre: *** cfs_fail_loc=228, val=0*** [ 1420.575725] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 16:49:32 (1746132572) [ 1420.981149] Lustre: *** cfs_fail_loc=228, val=0*** [ 1422.730391] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 16:49:35 (1746132575) [ 1433.304230] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ========================================================== 16:49:45 (1746132585) [ 1435.671008] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 16:49:48 (1746132588) [ 1510.104742] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 16:51:02 (1746132662) [ 1518.678174] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 16:51:11 (1746132671) [ 1543.909195] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 16:51:36 (1746132696) [ 1716.673584] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 16:54:28 (1746132868) [ 1746.790766] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 16:54:59 (1746132899) [ 1749.894431] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 16:55:02 (1746132902) [ 1790.554884] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 16:55:42 (1746132942) [ 1796.369583] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 16:55:48 (1746132948) [ 1799.994548] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 16:55:52 (1746132952) [ 1825.035467] Lustre: DEBUG MARKER: == sanity test 101m: read ahead for small file and last stripe of the file ========================================================== 16:56:17 (1746132977) [ 1829.535607] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 16:56:21 (1746132981) [ 1831.819924] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 16:56:24 (1746132984) [ 1836.181225] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 16:56:28 (1746132988) [ 1839.120011] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 16:56:31 (1746132991) [ 1860.584041] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 16:56:52 (1746133012) [ 1881.717322] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 16:57:13 (1746133033) [ 1882.588932] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 1883.374342] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 1884.204379] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 1884.940351] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 1887.959195] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 16:57:20 (1746133040) [ 1889.191987] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 1890.011218] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 1890.715825] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 1891.475616] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 1892.337317] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 1895.812480] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 16:57:27 (1746133047) [ 1899.311267] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 16:57:31 (1746133051) [ 1920.224483] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 16:57:52 (1746133072) [ 1924.053058] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 16:57:56 (1746133076) [ 1927.769459] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 16:57:59 (1746133079) [ 1931.010172] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 16:58:03 (1746133083) [ 1934.938316] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 16:58:06 (1746133086) [ 1938.398370] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 16:58:10 (1746133090) [ 1941.909730] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 16:58:13 (1746133093) [ 1946.288870] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 16:58:18 (1746133098) [ 1950.027770] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 16:58:22 (1746133102) [ 1953.562752] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 16:58:25 (1746133105) [ 2091.630603] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 17:00:43 (1746133243) [ 2171.374523] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 17:02:03 (1746133323) [ 2174.947293] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 17:02:06 (1746133326) [ 2668.874346] Lustre: lustre-MDT0000: Client aedcc92b-f8d3-439d-9a03-d2dac944b382 (at 192.168.202.47@tcp) reconnecting [ 2781.262705] Lustre: 7237:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 30372 > trans_max 3200 [ 2781.265718] Lustre: 7237:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 2781.267690] Lustre: 7237:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28148/0 [ 2781.273067] Lustre: 7237:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 2781.276063] Lustre: 7237:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 2781.279640] Lustre: 7237:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 2781.283521] CPU: 2 PID: 7237 Comm: mdt00_000 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 2781.286763] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 2781.289739] Call Trace: [ 2781.290632] [] dump_stack+0x19/0x1b [ 2781.292571] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 2781.297369] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 2781.301219] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 2781.303396] [] lod_trans_start+0x7f/0x2e0 [lod] [ 2781.305414] [] mdd_trans_start+0x14/0x20 [mdd] [ 2781.307462] [] mdd_unlink+0x617/0xea0 [mdd] [ 2781.309113] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 2781.311541] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 2781.316296] [] mdt_reint_rec+0x87/0x240 [mdt] [ 2781.320383] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 2781.324916] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 2781.329389] [] mdt_reint+0x67/0x150 [mdt] [ 2781.333030] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 2781.336951] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 2781.340269] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 2781.342746] [] ? put_prev_entity+0x31/0x400 [ 2781.345410] [] ? do_raw_spin_unlock+0x49/0x90 [ 2781.347895] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 2781.350443] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 2781.354239] [] kthread+0xe4/0xf0 [ 2781.357616] [] ? kthread_create_on_node+0x140/0x140 [ 2781.361792] [] ret_from_fork_nospec_begin+0x7/0x21 [ 2781.364130] [] ? kthread_create_on_node+0x140/0x140 [ 2785.367872] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 17:12:17 (1746133937) [ 2787.194074] Lustre: lustre-MDD0000: changelog on [ 2788.545087] Lustre: lustre-MDD0001: changelog on [ 2792.367964] Lustre: lustre-MDD0001: changelog off [ 2793.864268] Lustre: lustre-MDD0000: changelog off [ 2795.249187] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 17:12:27 (1746133947) [ 2795.597441] Lustre: lustre-OST0000: Client aedcc92b-f8d3-439d-9a03-d2dac944b382 (at 192.168.202.47@tcp) reconnecting [ 2797.383815] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88007cf97800.ost_server_uuid 50 [ 2798.158287] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88007cf97800.ost_server_uuid in FULL state after 0 sec [ 2802.171595] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 17:12:34 (1746133954) [ 2806.445834] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 17:12:38 (1746133958) [ 2807.301899] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [ 2808.538863] Lustre: DEBUG MARKER: == sanity test 104d: runas -u 500 -g 500 lctl dl test ==== 17:12:40 (1746133960) [ 2812.502351] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 17:12:44 (1746133964) [ 2816.765089] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 17:12:48 (1746133968) [ 2821.031073] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 17:12:52 (1746133972) [ 2825.560137] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 17:12:57 (1746133977) [ 2839.836261] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 17:13:11 (1746133991) [ 2843.944117] Lustre: DEBUG MARKER: == sanity test 105f: Enqueue same range flocks =========== 17:13:15 (1746133995) [ 2849.634899] Lustre: DEBUG MARKER: == sanity test 105g: ldlm_lock_debug stack test ========== 17:13:21 (1746134001) [ 2855.376686] Lustre: DEBUG MARKER: == sanity test 105h: Flock functional verify ============= 17:13:27 (1746134007) [ 2859.712493] Lustre: DEBUG MARKER: == sanity test 105i: Flock deadlock verify =============== 17:13:31 (1746134011) [ 2869.449539] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 17:13:41 (1746134021) [ 2874.235865] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 17:13:45 (1746134025) [ 2879.557964] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 17:13:51 (1746134031) [ 2883.594164] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 17:13:55 (1746134035) [ 3051.985016] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 17:16:43 (1746134203) [ 3053.912710] Lustre: *** cfs_fail_loc=147, val=0*** [ 3060.278261] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 17:16:52 (1746134212) [ 3065.225093] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 17:16:57 (1746134217) [ 3070.014912] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 17:17:01 (1746134221) [ 3071.359678] Lustre: *** cfs_fail_loc=217, val=0*** [ 3071.362648] Lustre: Skipped 7 previous similar messages [ 3076.731106] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 3078.082746] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 3079.419483] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 17:17:11 (1746134231) [ 3084.457140] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 17:17:16 (1746134236) [ 3089.500180] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 17:17:21 (1746134241) [ 3090.932823] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3091.979329] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3094.004614] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3096.656035] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3100.666322] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3106.608563] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 17:17:38 (1746134258) [ 3109.312660] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3109.315336] Lustre: Skipped 1 previous similar message [ 3118.710121] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 17:17:50 (1746134270) [ 3125.633461] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 17:17:57 (1746134277) [ 3126.960723] Lustre: *** cfs_fail_loc=20e, val=0*** [ 3126.962387] Lustre: Skipped 2 previous similar messages [ 3143.588545] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 17:18:15 (1746134295) [ 3149.467893] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 17:18:20 (1746134300) [ 3154.602464] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 17:18:26 (1746134306) [ 3158.217654] LustreError: 11159:0:(ofd_dev.c:1805:ofd_statfs_hdl()) cfs_fail_timeout id 242 sleeping for 10000ms [ 3158.540248] LustreError: 11159:0:(ofd_dev.c:1805:ofd_statfs_hdl()) cfs_fail_timeout interrupted [ 3163.185372] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 17:18:35 (1746134315) [ 3167.584030] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 17:18:39 (1746134319) [ 3172.348070] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 17:18:44 (1746134324) [ 3176.892863] Lustre: DEBUG MARKER: == sanity test 119e: Basic tests of dio read and write at various sizes ========================================================== 17:18:48 (1746134328) [ 3179.995459] Lustre: DEBUG MARKER: sanity test_119e: @@@@@@ FAIL: trivial unaligned dio failed [ 3186.318851] Lustre: DEBUG MARKER: == sanity test 119f: dio vs dio race ===================== 17:18:58 (1746134338) [ 3199.833258] Lustre: DEBUG MARKER: sanity test_119f: @@@@@@ FAIL: dio copy 1 w/bsize 1049600 failed: 1 [ 3205.366329] Lustre: DEBUG MARKER: == sanity test 119g: dio vs buffered I/O race ============ 17:19:17 (1746134357) [ 3223.082991] Lustre: DEBUG MARKER: sanity test_119g: @@@@@@ FAIL: dio copy 1 w/bsize 1049600 failed: 1 [ 3229.107643] Lustre: DEBUG MARKER: == sanity test 119h: basic tests of memory unaligned dio ========================================================== 17:19:40 (1746134380) [ 3230.983600] Lustre: DEBUG MARKER: sanity test_119h: @@@@@@ FAIL: multiop memory unaligned write failed, 1044480 [ 3236.018430] Lustre: DEBUG MARKER: SKIP: sanity test_119i skipping ALWAYS excluded test 119i [ 3237.466984] Lustre: DEBUG MARKER: == sanity test 119j: basic tests of hybrid IO switching == 17:19:49 (1746134389) [ 3238.801821] Lustre: DEBUG MARKER: SKIP: sanity test_119j needs kernel > 4.5.0 for ki_flags support [ 3240.267004] Lustre: DEBUG MARKER: == sanity test 119m: Test DIO readv/writev: exercise iter duplication ========================================================== 17:19:51 (1746134391) [ 3244.860462] Lustre: DEBUG MARKER: == sanity test 119n: Test Unaligned DIO readv() and writev() with unpatched ZFS ========================================================== 17:19:56 (1746134396) [ 3246.066270] Lustre: DEBUG MARKER: SKIP: sanity test_119n need ZFS server without unaligned_dio support [ 3247.396262] Lustre: DEBUG MARKER: == sanity test 119o: Test Unaligned DIO readv() and writev() with unpatched servers ========================================================== 17:19:59 (1746134399) [ 3248.612354] Lustre: DEBUG MARKER: SKIP: sanity test_119o need ldiskfs without unaligned_dio support. [ 3249.975809] Lustre: DEBUG MARKER: == sanity test 119p: Test Unaligned DIO readv() and writev() with patched servers ========================================================== 17:20:01 (1746134401) [ 3250.985779] Lustre: DEBUG MARKER: sanity test_119p: @@@@@@ FAIL: DIO unaligned writev test failed [ 3254.970753] Lustre: DEBUG MARKER: == sanity test 119q: Test patchded Unaligned DIO readv() and writev() ========================================================== 17:20:06 (1746134406) [ 3256.124039] Lustre: DEBUG MARKER: sanity test_119q: @@@@@@ FAIL: DIO unaligned writev test failed: 69631 [ 3259.518556] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 17:20:11 (1746134411) [ 3265.937834] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 17:20:17 (1746134417) [ 3272.226452] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 17:20:24 (1746134424) [ 3279.416474] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 17:20:31 (1746134431) [ 3285.038989] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 17:20:36 (1746134436) [ 3297.506071] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 17:20:49 (1746134449) [ 3311.310625] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 17:21:03 (1746134463) [ 3618.790405] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 17:26:10 (1746134770) [ 3623.402437] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 17:26:15 (1746134775) [ 3630.206207] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 2 sec [ 3631.891865] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 0 sec [ 3659.506817] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 16 sec [ 3663.671334] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 3 sec [ 3738.176206] Lustre: DEBUG MARKER: ls -l 3000 files without statahead: 50 sec [ 3747.479068] Lustre: DEBUG MARKER: ls -l 3000 files with statahead: 7 sec [ 3748.438807] Lustre: DEBUG MARKER: ls -l done [ 3786.589968] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity: 37 seconds [ 3791.561218] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 17:29:03 (1746134943) [ 3792.412276] Lustre: DEBUG MARKER: SKIP: sanity test_123ab Test must be statx() syscall supported [ 3793.565548] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 17:29:05 (1746134945) [ 3794.497890] Lustre: DEBUG MARKER: SKIP: sanity test_123ac Test must be statx() syscall supported [ 3795.729652] Lustre: DEBUG MARKER: == sanity test 123ad: Verify batching statahead works correctly ========================================================== 17:29:07 (1746134947) [ 3802.846337] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 2 sec [ 3804.408995] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 3835.254647] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 17 sec [ 3841.729620] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 5 sec [ 3892.237893] Lustre: DEBUG MARKER: ls -l 2000 files without statahead: 36 sec [ 3904.196624] Lustre: DEBUG MARKER: ls -l 2000 files with statahead: 11 sec [ 3905.183568] Lustre: DEBUG MARKER: ls -l done [ 3935.362425] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 28 seconds [ 4349.820438] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 2 sec [ 4351.535264] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 4379.568511] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 15 sec [ 4383.540073] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 2 sec [ 4450.709467] Lustre: DEBUG MARKER: ls -l 3000 files without statahead: 45 sec [ 4458.109734] Lustre: DEBUG MARKER: ls -l 3000 files with statahead: 6 sec [ 4458.930582] Lustre: DEBUG MARKER: ls -l done [ 4490.324738] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 31 seconds [ 4757.120270] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 17:45:09 (1746135909) [ 4771.351898] Lustre: DEBUG MARKER: ls done [ 4787.129082] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 17:45:39 (1746135939) [ 4792.777851] Lustre: DEBUG MARKER: == sanity test 123d: Statahead on striped directories works correctly ========================================================== 17:45:44 (1746135944) [ 4805.410545] Lustre: DEBUG MARKER: == sanity test 123e: statahead with large wide striping == 17:45:57 (1746135957) [ 4866.900952] Lustre: 7243:0:(mdt_handler.c:4672:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 4917.342570] Lustre: DEBUG MARKER: == sanity test 123f: Retry mechanism with large wide striping files ========================================================== 17:47:49 (1746136069) [ 4963.123900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4963.130204] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5047.324349] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5047.773199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5118.214253] Lustre: 7243:0:(mdt_handler.c:4672:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 5168.241223] Lustre: 9432:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 19632 > trans_max 3200 [ 5168.246153] Lustre: 9432:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 5168.251840] Lustre: 9432:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1287/1287/0, xattr_set: 1930/18124/0 [ 5168.258961] Lustre: 9432:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 5168.263140] Lustre: 9432:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 5168.267371] Lustre: 9432:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 5168.272113] CPU: 2 PID: 9432 Comm: mdt00_008 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 5168.276561] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 5168.279592] Call Trace: [ 5168.280551] [] dump_stack+0x19/0x1b [ 5168.282463] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 5168.284973] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 5168.289816] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 5168.293505] [] lod_trans_start+0x7f/0x2e0 [lod] [ 5168.295382] [] mdd_trans_start+0x14/0x20 [mdd] [ 5168.298156] [] mdd_unlink+0x617/0xea0 [mdd] [ 5168.300257] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 5168.303461] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 5168.305966] [] mdt_reint_rec+0x87/0x240 [mdt] [ 5168.308957] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 5168.311589] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 5168.313740] [] mdt_reint+0x67/0x150 [mdt] [ 5168.316512] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 5168.319034] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 5168.321576] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 5168.324950] [] ? do_raw_spin_unlock+0x49/0x90 [ 5168.328119] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 5168.330978] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 5168.333440] [] kthread+0xe4/0xf0 [ 5168.335424] [] ? kthread_create_on_node+0x140/0x140 [ 5168.339686] [] ret_from_fork_nospec_begin+0x7/0x21 [ 5168.343954] [] ? kthread_create_on_node+0x140/0x140 [ 5219.260432] Lustre: DEBUG MARKER: == sanity test 123g: Test for stat-ahead advise ========== 17:52:51 (1746136371) [ 5242.054977] Lustre: DEBUG MARKER: == sanity test 123h: Verify statahead work with the fname pattern via du ========================================================== 17:53:13 (1746136393) [ 5992.164480] Lustre: DEBUG MARKER: == sanity test 123i: Verify statahead work with the fname indexing pattern ========================================================== 18:05:44 (1746137144) [ 6076.366977] Lustre: DEBUG MARKER: == sanity test 123j: -ENOENT error from batched statahead be handled correctly ========================================================== 18:07:08 (1746137228) [ 6080.186546] Lustre: DEBUG MARKER: == sanity test 123k: Verify statahead work with mdtest shared stat() mode ========================================================== 18:07:12 (1746137232) [ 6080.997728] Lustre: DEBUG MARKER: SKIP: sanity test_123k mdtest not found [ 6081.997743] Lustre: DEBUG MARKER: == sanity test 123l: Avoid panic when revalidate a local cached entry ========================================================== 18:07:14 (1746137234) [ 6154.945652] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 18:08:27 (1746137307) [ 6155.577882] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 6173.821909] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d74000 [ 6174.458294] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8d74000 [ 6175.144177] Lustre: DEBUG MARKER: LRU=1015 [ 6175.652649] Lustre: DEBUG MARKER: LIMIT=46624 [ 6176.214167] Lustre: DEBUG MARKER: LVF=5512100 [ 6176.793029] Lustre: DEBUG MARKER: OLD_LVF=100 [ 6177.379612] Lustre: DEBUG MARKER: Sleep 50 sec [ 6228.199220] Lustre: DEBUG MARKER: Dropped 430 locks in 50s [ 6228.899357] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 6239.287680] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 18:09:51 (1746137391) [ 6286.807821] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 6353.313711] Lustre: DEBUG MARKER: ls -la time: 66 seconds [ 6353.808981] Lustre: DEBUG MARKER: lru_size = 400 [ 6439.227957] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 6468.536401] Lustre: DEBUG MARKER: ls -la time: 27 seconds [ 6469.246823] Lustre: DEBUG MARKER: lru_size = 4052 [ 6469.905696] Lustre: DEBUG MARKER: ls -la is 59% faster with lru resize enabled [ 6501.074303] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 18:14:13 (1746137653) [ 6526.642335] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize disabled ========================================================== 18:14:38 (1746137678) [ 6551.600867] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 18:15:03 (1746137703) [ 6554.472486] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 18:15:06 (1746137706) [ 6557.499766] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 18:15:09 (1746137709) [ 6561.195478] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 18:15:13 (1746137713) [ 6564.720063] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [ 6581.372824] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 18:15:33 (1746137733) [ 6584.962872] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 18:15:37 (1746137737) [ 6594.383711] Lustre: 9432:0:(osd_handler.c:600:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513239, FID: [0x20000040b:0x8319:0x0]) is approaching max size limit [ 6596.082385] Lustre: 7237:0:(osd_handler.c:600:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513239, FID: [0x20000040b:0x8319:0x0]) is approaching max size limit [ 6606.355683] Lustre: 7237:0:(osd_handler.c:596:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513239, FID: [0x20000040b:0x8319:0x0]) has reached max size limit [ 6625.420080] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 18:16:17 (1746137777) [ 6629.007613] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 18:16:21 (1746137781) [ 6633.615268] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 18:16:25 (1746137785) [ 6637.441638] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 18:16:29 (1746137789) [ 6638.308105] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [ 6639.447851] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 18:16:31 (1746137791) [ 6672.852994] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 18:17:04 (1746137824) [ 6676.282934] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 18:17:08 (1746137828) [ 6689.729508] Lustre: 9406:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 6372 > trans_max 3200 [ 6689.732491] Lustre: 9406:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 6689.738229] Lustre: 9406:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 403/403/0, xattr_set: 604/5748/0 [ 6689.743493] Lustre: 9406:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 6689.747839] Lustre: 9406:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 6689.752577] Lustre: 9406:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 6689.755817] CPU: 3 PID: 9406 Comm: mdt00_007 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 6689.759203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 6689.761423] Call Trace: [ 6689.761867] [] dump_stack+0x19/0x1b [ 6689.762994] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 6689.766150] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 6689.769420] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 6689.771672] [] lod_trans_start+0x7f/0x2e0 [lod] [ 6689.774407] [] mdd_trans_start+0x14/0x20 [mdd] [ 6689.776487] [] mdd_unlink+0x617/0xea0 [mdd] [ 6689.778382] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 6689.780553] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 6689.782977] [] mdt_reint_rec+0x87/0x240 [mdt] [ 6689.785173] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 6689.788606] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 6689.792790] [] mdt_reint+0x67/0x150 [mdt] [ 6689.797544] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 6689.803158] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 6689.809196] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 6689.812845] [] ? put_prev_entity+0x31/0x400 [ 6689.815964] [] ? do_raw_spin_unlock+0x49/0x90 [ 6689.819130] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 6689.821171] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 6689.824256] [] kthread+0xe4/0xf0 [ 6689.826587] [] ? kthread_create_on_node+0x140/0x140 [ 6689.830133] [] ret_from_fork_nospec_begin+0x7/0x21 [ 6689.833827] [] ? kthread_create_on_node+0x140/0x140 [ 6692.627175] Lustre: DEBUG MARKER: == sanity test 130h: FIEMAP deadlock ===================== 18:17:24 (1746137844) [ 6702.190026] Lustre: DEBUG MARKER: == sanity test 130i: FIEMAP (DoM file) =================== 18:17:34 (1746137854) [ 6724.409303] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 18:17:56 (1746137876) [ 6727.853568] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 18:17:59 (1746137879) [ 6731.852746] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 18:18:03 (1746137883) [ 6734.879350] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 18:18:07 (1746137887) [ 6737.981460] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 18:18:10 (1746137890) [ 6740.585407] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 18:18:12 (1746137892) [ 6749.392016] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 18:18:21 (1746137901) [ 6753.957557] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 18:18:26 (1746137906) [ 6771.339363] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 18:18:43 (1746137923) [ 6787.854117] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 18:19:00 (1746137940) [ 6793.554428] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 18:19:05 (1746137945) [ 6804.441403] LustreError: 24632:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6804.443953] LustreError: 24632:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6804.540934] Lustre: server umount lustre-MDT0000 complete [ 6805.717355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6805.720404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6805.726713] LustreError: 7112:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.689964] LustreError: 10932:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746137960 with bad export cookie 17536376764221600650 [ 6807.694121] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6807.699463] LustreError: 10932:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6807.780654] LustreError: 25229:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6807.783184] LustreError: 25229:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6807.899242] Lustre: server umount lustre-MDT0001 complete [ 6821.662930] LustreError: 25822:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6821.665795] LustreError: 25822:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6821.732279] Lustre: server umount lustre-OST0000 complete [ 6834.991279] LustreError: 26424:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6834.995210] LustreError: 26424:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6835.173183] Lustre: server umount lustre-OST0001 complete [ 6837.589376] device-mapper: core: cleaned up [ 6840.816823] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 6841.460973] Key type lgssc unregistered [ 6841.557313] LNet: 27288:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6842.567069] LNet: Removed LNI 192.168.202.147@tcp [ 6848.743519] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 6848.964363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6849.036215] alg: No test for adler32 (adler32-zlib) [ 6849.906124] Lustre: Lustre: Build Version: 2.16.54_84_g4581c23 [ 6850.007216] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 6850.010341] LNet: Accept secure, port 988 [ 6851.572335] Key type lgssc registered [ 6851.989916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6858.069184] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 6859.968810] device-mapper: uevent: version 1.0.3 [ 6859.971157] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 6863.134030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6863.144519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6864.414815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6865.549349] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [ 6869.839377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6869.914494] Lustre: 30566:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6870.094999] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6870.153032] mount.lustre (30566) used greatest stack depth: 9520 bytes left [ 6871.493959] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [ 6872.752759] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 6876.582317] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 6876.589484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6876.707156] Lustre: 31691:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6876.802632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6876.834212] mount.lustre (31691) used greatest stack depth: 9504 bytes left [ 6878.896131] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [ 6879.800898] LustreError: 31697:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6879.821430] LustreError: 31697:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6879.827360] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:10396 to 0x280000400:10433) [ 6884.307888] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 6884.315865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6884.376844] Lustre: 418:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6884.482419] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6886.786743] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [ 6889.490556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:21558 to 0x280000403:21633) [ 6889.492896] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:22084 to 0x2c0000403:22273) [ 6889.498535] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:10554 to 0x2c0000400:10593) [ 6895.809748] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6897.320374] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 6905.977763] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 18:20:58 (1746138058) [ 6925.947409] LNet: 3180:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 6925.970530] Lustre: DEBUG MARKER:  [ 6925.971858] Lustre: DEBUG MARKER:  [ 6947.828299] LNet: 4802:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 6947.838310] LNet: 4802:0:(debug.c:388:cfs_str2mask()) Skipped 3 previous similar messages [ 6947.859451] Lustre: DEBUG MARKER:  [ 6947.862911] Lustre: DEBUG MARKER:  [ 6954.601486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6954.602950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6954.615164] Lustre: Skipped 3 previous similar messages [ 6956.829898] LustreError: 5078:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6956.885736] Lustre: server umount lustre-MDT0000 complete [ 6959.605735] LustreError: 29501:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6959.615328] LustreError: 29501:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6961.037376] LustreError: 29483:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746138114 with bad export cookie 10186573901395518515 [ 6961.045744] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6961.070308] LustreError: 29483:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6961.141364] LustreError: 5680:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6961.144503] LustreError: 5680:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6961.258586] Lustre: server umount lustre-MDT0001 complete [ 6975.859936] LustreError: 6274:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6975.862604] LustreError: 6274:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6975.905713] Lustre: server umount lustre-OST0000 complete [ 6989.838031] LustreError: 6877:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6989.840530] LustreError: 6877:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6989.943639] Lustre: server umount lustre-OST0001 complete [ 6992.801844] device-mapper: core: cleaned up [ 6996.082476] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing unload_modules_local [ 6997.094382] Key type lgssc unregistered [ 6997.205004] LNet: 7805:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6998.216874] LNet: Removed LNI 192.168.202.147@tcp [ 7004.557365] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 7004.796528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7004.815894] alg: No test for adler32 (adler32-zlib) [ 7005.689828] Lustre: Lustre: Build Version: 2.16.54_84_g4581c23 [ 7005.787686] LNet: Added LNI 192.168.202.147@tcp [8/256/0/180] [ 7005.789199] LNet: Accept secure, port 988 [ 7007.348511] Key type lgssc registered [ 7007.787472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7014.258803] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_modules_local [ 7016.294621] device-mapper: uevent: version 1.0.3 [ 7016.297951] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 7020.027103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7020.042599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7021.356917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7022.954755] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [ 7027.820528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7027.902310] Lustre: 11089:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7028.094773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7029.522484] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [ 7030.974648] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 7035.396320] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 7035.404831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7035.514588] Lustre: 12219:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7035.625899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7037.822394] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [ 7038.625594] LustreError: 12224:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7038.634011] LustreError: 12224:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 7038.644512] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:10396 to 0x280000400:10465) [ 7043.460911] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 7043.469367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7043.643890] LustreError: 12742:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.647555] Lustre: 13412:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7043.780853] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7045.790800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:21558 to 0x280000403:21665) [ 7045.794844] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:10554 to 0x2c0000400:10625) [ 7045.854567] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:22084 to 0x2c0000403:22305) [ 7046.385685] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [ 7051.705399] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7053.549081] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 7061.500583] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 18:23:33 (1746138213) [ 7535.340906] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 18:31:27 (1746138687) [ 7543.143176] Lustre: *** cfs_fail_loc=327, val=500*** [ 7563.284826] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 18:31:55 (1746138715) [ 7566.048394] Lustre: *** cfs_fail_loc=328, val=500*** [ 7566.050570] Lustre: Skipped 1308 previous similar messages [ 7567.052818] Lustre: *** cfs_fail_loc=328, val=500*** [ 7567.055285] Lustre: Skipped 162 previous similar messages [ 7567.623010] Lustre: 10015:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8800859a4380 x1830958560007424/t0(0) o101->21f0fbb0-791d-4c74-ac36-2c88602f9051@192.168.202.47@tcp:401/0 lens 648/0 e 0 to 0 dl 1746138731 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7568.648100] Lustre: 10015:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8800859a6680 x1830958560008192/t0(0) o101->21f0fbb0-791d-4c74-ac36-2c88602f9051@192.168.202.47@tcp:402/0 lens 648/0 e 0 to 0 dl 1746138732 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7569.677013] Lustre: *** cfs_fail_loc=328, val=500*** [ 7569.678731] Lustre: Skipped 90 previous similar messages [ 7570.396969] Lustre: 10014:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff880136901f80 x1830958560072960/t0(0) o101->21f0fbb0-791d-4c74-ac36-2c88602f9051@192.168.202.47@tcp:404/0 lens 648/0 e 0 to 0 dl 1746138734 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7572.427588] Lustre: 10015:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff880136901500 x1830958560074496/t0(0) o101->21f0fbb0-791d-4c74-ac36-2c88602f9051@192.168.202.47@tcp:406/0 lens 648/0 e 0 to 0 dl 1746138736 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7572.442738] Lustre: 10015:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) Skipped 1 previous similar message [ 7574.484508] Lustre: *** cfs_fail_loc=328, val=500*** [ 7574.490063] Lustre: Skipped 127 previous similar messages [ 7576.881450] Lustre: 11577:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8800a510f100 x1830958560108160/t0(0) o101->21f0fbb0-791d-4c74-ac36-2c88602f9051@192.168.202.47@tcp:411/0 lens 648/0 e 0 to 0 dl 1746138741 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7576.898648] Lustre: 11577:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) Skipped 3 previous similar messages [ 7583.169257] Lustre: *** cfs_fail_loc=328, val=500*** [ 7583.170892] Lustre: Skipped 93 previous similar messages [ 7585.204225] Lustre: 10014:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff880098963800 x1830958560129024/t0(0) o101->21f0fbb0-791d-4c74-ac36-2c88602f9051@192.168.202.47@tcp:419/0 lens 648/0 e 0 to 0 dl 1746138749 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [ 7585.216449] Lustre: 10014:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) Skipped 7 previous similar messages [ 7586.485592] LustreError: 22291:0:(ldlm_lockd.c:3258:lock_reclaim_threshold_mb_store()) Failed to set lock_reclaim_threshold_mb, rc = -22. [ 7594.882432] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [ 7595.739718] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [ 7597.009978] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 18:32:28 (1746138748) [ 7620.948031] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 18:32:52 (1746138772) [ 7643.970659] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 18:33:16 (1746138796) [ 7659.130511] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 18:33:31 (1746138811) [ 7679.042001] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 18:33:51 (1746138831) [ 7688.424175] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 18:34:00 (1746138840) [ 7698.546876] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 18:34:10 (1746138850) [ 7712.244975] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 18:34:24 (1746138864) [ 7723.598648] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 18:34:35 (1746138875) [ 7733.927374] Lustre: DEBUG MARKER: == sanity test 150h: Verify extend fallocate updates the file size ========================================================== 18:34:46 (1746138886) [ 7734.878128] Lustre: DEBUG MARKER: SKIP: sanity test_150h Test must be statx() syscall supported [ 7735.762223] Lustre: DEBUG MARKER: == sanity test 150ia: Verify fallocate zero-range ZERO functionality ========================================================== 18:34:47 (1746138887) [ 7744.588409] Lustre: DEBUG MARKER: == sanity test 150ib: Verify fallocate zero-range PREALLOC functionality ========================================================== 18:34:56 (1746138896) [ 7754.238570] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 18:35:06 (1746138906) [ 7760.520236] bash (32335): drop_caches: 1 [ 7765.996162] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 18:35:18 (1746138918) [ 7768.950149] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 18:35:21 (1746138921) [ 7771.825096] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 18:35:24 (1746138924) [ 7774.633272] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 18:35:26 (1746138926) [ 7777.701039] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 18:35:29 (1746138929) [ 7778.434918] LustreError: 10013:0:(fld_handler.c:244:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 7782.540410] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 18:35:34 (1746138934) [ 7783.289563] LustreError: 31905:0:(fld_handler.c:244:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 7783.298423] LustreError: 31905:0:(fld_handler.c:244:fld_server_lookup()) Skipped 3 previous similar messages [ 7787.772203] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 18:35:39 (1746138939) [ 7790.917390] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 18:35:43 (1746138943) [ 7794.978869] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 18:35:46 (1746138946) [ 7798.271370] Lustre: DEBUG MARKER: == sanity test 154ea: .lustre is not returned by readdir (2) ========================================================== 18:35:50 (1746138950) [ 7830.304427] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 18:36:22 (1746138982) [ 7834.203149] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 18:36:26 (1746138986) [ 8295.179516] Lustre: DEBUG MARKER: == sanity test 154h: Verify interactive path2fid ========= 18:44:07 (1746139447) [ 8298.067381] Lustre: DEBUG MARKER: == sanity test 154i: fid2path for path longer than PATH_MAX ========================================================== 18:44:10 (1746139450) [ 8315.939345] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 18:44:28 (1746139468) [ 8320.910634] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 18:44:33 (1746139473) [ 8326.169573] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 18:44:38 (1746139478) [ 8331.587196] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 18:44:43 (1746139483) [ 8338.384090] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 18:44:50 (1746139490) [ 8370.847079] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 18:45:22 (1746139522) [ 8398.974238] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 18:45:51 (1746139551) [ 8426.530644] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 18:46:18 (1746139578) [ 8457.320063] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 18:46:49 (1746139609) [ 8461.899495] Lustre: DEBUG MARKER: Turn on read and write cache [ 8463.687676] Lustre: DEBUG MARKER: Write data and read it back. [ 8464.284549] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 8466.043869] Lustre: DEBUG MARKER: cache hits: before: 65582, after: 65585 [ 8466.646759] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8467.858719] Lustre: DEBUG MARKER: cache hits:: before: 65585, after: 65588 [ 8468.517660] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [ 8470.177614] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8472.200281] Lustre: DEBUG MARKER: cache hits:: before: 65588, after: 65591 [ 8472.899813] Lustre: DEBUG MARKER: Write data and read it back. [ 8473.676999] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 8475.601732] Lustre: DEBUG MARKER: cache hits:: before: 65591, after: 65594 [ 8476.210301] Lustre: DEBUG MARKER: Turn off read and write cache [ 8477.713370] Lustre: DEBUG MARKER: Write data and read it back [ 8478.361616] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 8479.939611] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65594 [ 8480.547789] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [ 8482.051476] Lustre: DEBUG MARKER: Write data and read it back [ 8482.689074] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 8484.517864] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65594 [ 8485.127640] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 8486.865617] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65597 [ 8490.253157] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 18:47:22 (1746139642) [ 8491.314303] Lustre: lustre-MDD0000: changelog on [ 8492.484188] Lustre: lustre-MDD0001: changelog on [ 8500.130615] Lustre: Failing over lustre-MDT0000 [ 8500.261153] LustreError: 17091:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8500.345206] Lustre: server umount lustre-MDT0000 complete [ 8500.566136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8500.573420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.582649] LustreError: 11115:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8502.716171] LustreError: 10014:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8503.320277] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8503.329149] Lustre: Skipped 2 previous similar messages [ 8504.418916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8504.495588] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8504.668456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8504.698138] Lustre: lustre-MDD0000: changelog on [ 8504.709130] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8506.482501] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [ 8507.702203] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8509.658987] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [ 8509.855206] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 8509.879030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:22503 to 0x280000403:22529) [ 8509.879158] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23141 to 0x2c0000403:23169) [ 8510.243443] Lustre: lustre-MDD0000: changelog off [ 8511.065540] Lustre: lustre-MDD0001: changelog off [ 8516.636743] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 18:47:48 (1746139668) [ 8517.746616] Lustre: lustre-MDD0000: changelog on [ 8522.612329] Lustre: lustre-MDD0001: changelog off [ 8524.702171] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 18:47:56 (1746139676) [ 8525.844849] Lustre: lustre-MDD0000: changelog on [ 8525.846612] Lustre: Skipped 1 previous similar message [ 8530.897435] Lustre: lustre-MDD0001: changelog off [ 8530.900261] Lustre: Skipped 1 previous similar message [ 8533.212175] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 18:48:05 (1746139685) [ 8534.642607] Lustre: lustre-MDD0000: changelog on [ 8534.644045] Lustre: Skipped 1 previous similar message [ 8539.453975] Lustre: lustre-MDD0001: changelog off [ 8539.456691] Lustre: Skipped 1 previous similar message [ 8541.672047] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 18:48:13 (1746139693) [ 8548.162639] Lustre: lustre-MDD0000: changelog off [ 8548.164518] Lustre: Skipped 2 previous similar messages [ 8549.251589] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 18:48:21 (1746139701) [ 8551.920883] Lustre: lustre-MDD0001: changelog on [ 8551.923860] Lustre: Skipped 4 previous similar messages [ 8555.099995] Lustre: DEBUG MARKER: 1746139707: creating first dirs [ 8571.450903] Lustre: *** cfs_fail_loc=1313, val=3*** [ 8571.452740] Lustre: 11577:0:(mdd_dir.c:911:mdd_changelog_emrg_cleanup()) lustre-MDD0000: changelog has only 3 free catalog entries [ 8571.457892] Lustre: 11577:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 8571.464081] Lustre: 25579:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl7 idle for 18s with 4 unprocessed records [ 8582.138002] Lustre: lustre-MDD0001: changelog off [ 8585.193027] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 18:48:57 (1746139737) [ 8586.802089] Lustre: lustre-MDD0000: changelog on [ 8598.350681] Lustre: 10014:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 8598.354079] Lustre: 10014:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 8598.360970] Lustre: 28873:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl9 idle for 8s with 4 unprocessed records [ 8598.367048] Lustre: 28873:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 8610.981426] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 18:49:23 (1746139763) [ 8632.734314] Lustre: *** cfs_fail_loc=1316, val=0*** [ 8632.737440] Lustre: 31905:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0001: simulate starting changelog garbage collection [ 8632.745144] Lustre: 31905:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 8632.749979] Lustre: 32164:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0001: force deregister of changelog user cl11 idle for 18s with 3 unprocessed records [ 8632.761659] Lustre: 32164:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 8633.731926] Lustre: Failing over lustre-MDT0000 [ 8633.805249] Lustre: Failing over lustre-MDT0001 [ 8634.856441] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8634.857415] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8634.865059] Lustre: Skipped 2 previous similar messages [ 8636.769131] LustreError: 32480:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 8636.772229] LustreError: 32480:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8636.813456] Lustre: server umount lustre-MDT0001 complete [ 8636.895842] Lustre: server umount lustre-MDT0000 complete [ 8640.229041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8640.308204] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8640.485113] LustreError: 12234:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8640.495075] LustreError: 12234:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 8640.551127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8640.579813] Lustre: 1276:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 28 to index 29, being cleared now [ 8640.595683] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8641.875499] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [ 8642.950934] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8642.959540] LustreError: 1287:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8645.439380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8645.496446] Lustre: 2164:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8645.557287] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8645.560376] Lustre: Skipped 3 previous similar messages [ 8645.593013] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 8645.689799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8645.715634] Lustre: 2164:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0001 : orphan changelog records found, starting from index 18 to index 19, being cleared now [ 8645.736365] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8647.118635] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [ 8647.669255] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [ 8647.805304] Lustre: lustre-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 8647.834047] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:10472 to 0x280000400:10497) [ 8647.835766] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:10631 to 0x2c0000400:10657) [ 8650.785191] Lustre: lustre-MDT0000: Recovery over after 0:08, of 2 clients 2 recovered and 0 were evicted. [ 8650.787474] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [ 8650.787475] Lustre: Skipped 3 previous similar messages [ 8650.819499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:22503 to 0x280000403:22561) [ 8650.820283] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23171 to 0x2c0000403:23201) [ 8655.685571] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [ 8658.016841] Lustre: lustre-MDD0001: changelog off [ 8658.021118] Lustre: Skipped 3 previous similar messages [ 8660.639486] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 18:50:12 (1746139812) [ 8662.104218] Lustre: lustre-MDD0000: changelog on [ 8662.106951] Lustre: Skipped 5 previous similar messages [ 8665.881602] LustreError: 4716:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_race id 1315 sleeping [ 8667.957955] LustreError: 4886:0:(mdd_device.c:1746:mdd_changelog_user_register()) cfs_fail_race id 1315 waking [ 8667.962935] LustreError: 4716:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_fail_race id 1315 awake: rc=2923 [ 8668.755969] LustreError: 5142:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_fail_race id 1315 waking [ 8679.828291] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 18:50:31 (1746139831) [ 8692.807859] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 18:50:44 (1746139844) [ 8696.554731] LustreError: 1287:0:(mdd_dir.c:966:mdd_changelog_store()) cfs_fail_timeout id 15d sleeping for 3000ms [ 8699.565800] LustreError: 1287:0:(mdd_dir.c:966:mdd_changelog_store()) cfs_fail_timeout id 15d awake [ 8709.863272] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 18:51:01 (1746139861) [ 8722.577314] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 18:51:14 (1746139874) [ 8730.670085] LustreError: 1286:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_race id 15f sleeping [ 8732.676761] LustreError: 2782:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_fail_race id 15f waking [ 8732.685346] LustreError: 2782:0:(mdd_device.c:394:llog_changelog_cancel_cb()) Skipped 1 previous similar message [ 8732.692240] LustreError: 1286:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_fail_race id 15f awake: rc=2984 [ 8738.720598] Lustre: lustre-MDD0001: changelog off [ 8738.722732] Lustre: Skipped 11 previous similar messages [ 8741.108679] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 18:51:33 (1746139893) [ 9907.616183] LustreError: 1285:0:(mdd_device.c:408:llog_changelog_cancel_cb()) cfs_race id 16c sleeping [ 9909.627927] LustreError: 2782:0:(llog_osd.c:1136:llog_osd_next_block()) cfs_fail_race id 16c waking [ 9909.636437] LustreError: 1285:0:(mdd_device.c:408:llog_changelog_cancel_cb()) cfs_fail_race id 16c awake: rc=2985 [ 9916.659553] Lustre: lustre-MDD0001: changelog off [ 9916.661203] Lustre: Skipped 1 previous similar message [ 9919.410118] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 19:11:11 (1746141071) [ 9920.757908] Lustre: lustre-MDD0000: changelog on [ 9920.759810] Lustre: Skipped 13 previous similar messages [ 9922.530150] LustreError: 16062:0:(mdd_device.c:1700:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [ 9922.983974] Lustre: 16126:0:(mdd_device.c:1717:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [ 9923.417503] LustreError: 16191:0:(mdd_device.c:1709:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [ 9936.684147] Lustre: lustre-MDD0000: changelog off [ 9936.685930] Lustre: Skipped 1 previous similar message [ 9940.225808] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 19:11:32 (1746141092) [ 9941.650516] Lustre: lustre-MDD0000: changelog on [ 9941.652263] Lustre: Skipped 1 previous similar message [ 9945.405782] Lustre: Failing over lustre-MDT0000 [ 9945.507859] LustreError: 18645:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9945.510472] LustreError: 18645:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 9945.583382] Lustre: server umount lustre-MDT0000 complete [ 9948.006162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9948.011101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9948.017638] Lustre: Skipped 1 previous similar message [ 9948.053171] LustreError: 12234:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9949.499925] LustreError: 1287:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9949.879460] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9949.887207] Lustre: Skipped 2 previous similar messages [ 9950.316502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9950.402321] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9950.591116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9950.631632] Lustre: 19308:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 90152 to index 18446744073709551615, being cleared now [ 9950.649462] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9952.059005] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [ 9954.487804] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9955.591004] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [ 9955.595325] Lustre: Skipped 3 previous similar messages [ 9955.607257] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9955.629158] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23203 to 0x2c0000403:23233) [ 9955.629725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:22563 to 0x280000403:22593) [ 9961.052511] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 19:11:53 (1746141113) [ 9966.451946] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records * time ========================================================== 19:11:58 (1746141118) [ 9976.241601] Lustre: 12996:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 9976.245714] Lustre: 12996:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [ 9976.250588] Lustre: 21897:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl2 idle for 864008s with 500000004 unprocessed records [ 9976.257917] Lustre: lustre-MDD0000: changelog off [ 9976.260020] Lustre: Skipped 3 previous similar messages [ 9987.189032] Lustre: DEBUG MARKER: == sanity test 160t: changelog garbage collect on lack of space ========================================================== 19:12:18 (1746141138) [ 9989.216905] Lustre: lustre-MDD0000: changelog on [ 9989.218463] Lustre: Skipped 4 previous similar messages [10025.240825] Lustre: *** cfs_fail_loc=18c, val=1211180*** [10025.247303] Lustre: Skipped 1 previous similar message [10025.257337] Lustre: 14980:0:(mdd_dir.c:881:mdd_changelog_is_space_safe()) lustre-MDD0000: changelog uses 21MB with 1MB space limit [10025.261266] Lustre: 14980:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [10025.267516] Lustre: 14980:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [10025.275985] Lustre: 24563:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl3-user1 idle for 36s with 7506 unprocessed records [10025.291101] Lustre: 24563:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [10025.837760] Lustre: *** cfs_fail_loc=18c, val=1211180*** [10047.226825] Lustre: lustre-MDD0001: changelog off [10047.229071] Lustre: Skipped 2 previous similar messages [10050.065097] Lustre: DEBUG MARKER: == sanity test 160u: changelog rename record type name and sname strings are correct ========================================================== 19:13:21 (1746141201) [10054.280319] Lustre: lustre-MDD0000: changelog on [10054.282169] Lustre: Skipped 1 previous similar message [10065.707458] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 19:13:37 (1746141217) [10087.919239] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 19:13:59 (1746141239) [10111.324414] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 19:14:23 (1746141263) [10125.871212] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 19:14:37 (1746141277) [10136.947650] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 19:14:49 (1746141289) [10140.550588] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 19:14:52 (1746141292) [10145.070231] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 19:14:57 (1746141297) [10203.955519] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 19:15:56 (1746141356) [10210.349251] Lustre: Failing over lustre-OST0000 [10210.417768] LustreError: 31804:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10210.421820] LustreError: 31804:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10210.448524] Lustre: server umount lustre-OST0000 complete [10211.237803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10211.243700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10211.253439] LustreError: 21390:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10211.265506] LustreError: 21390:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [10213.592700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10213.603145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10213.617030] LustreError: 21390:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10218.647978] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10218.653004] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10218.713784] Lustre: 475:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [10218.788168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10218.793492] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10219.932645] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10219.991336] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10219.991757] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [10219.991758] Lustre: Skipped 3 previous similar messages [10220.884207] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [10222.542266] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 19:16:14 (1746141374) [10248.454805] Lustre: Failing over lustre-OST0000 [10248.475747] LustreError: 1902:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10248.479114] LustreError: 1902:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10248.516819] Lustre: server umount lustre-OST0000 complete [10248.823836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10248.825306] LustreError: 21390:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10248.825308] LustreError: 21390:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [10248.847071] Lustre: Skipped 1 previous similar message [10252.669356] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10252.675538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10252.736061] Lustre: 2509:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [10252.853537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10252.870414] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10254.533706] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10254.849694] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10254.851058] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [10254.851059] Lustre: Skipped 1 previous similar message [10255.617224] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [10257.535095] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 19:16:49 (1746141409) [10273.537465] Lustre: Failing over lustre-OST0000 [10274.380604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [10274.384158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10274.384564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10274.396253] LustreError: Skipped 1 previous similar message [10275.032047] Lustre: lustre-OST0000: Not available for connect from 192.168.202.47@tcp (stopping) [10275.036831] Lustre: Skipped 1 previous similar message [10275.575806] LustreError: 3888:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10275.583295] LustreError: 3888:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10275.609978] Lustre: server umount lustre-OST0000 complete [10277.897563] LustreError: 21392:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10277.913092] LustreError: 21392:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [10279.383428] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10279.394134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10279.465152] Lustre: 4485:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [10279.553448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10279.564806] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10280.026989] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10281.529536] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.147@tcp (at 0@lo) [10281.529967] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10281.537803] Lustre: Skipped 1 previous similar message [10281.656335] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [10283.284855] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 19:17:15 (1746141435) [10307.185510] Lustre: Failing over lustre-OST0000 [10307.209037] LustreError: 6923:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10307.215789] LustreError: 6923:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10307.241363] Lustre: server umount lustre-OST0000 complete [10309.592134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10309.595019] LustreError: 12742:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.606282] Lustre: Skipped 2 previous similar messages [10310.842087] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10310.850923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10310.911917] Lustre: 7518:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [10310.988970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10310.997690] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10312.275650] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10312.555972] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10312.556146] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.147@tcp (at 0@lo) [10312.556148] Lustre: Skipped 1 previous similar message [10313.035588] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [10314.812363] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 19:17:46 (1746141466) [10328.040621] Lustre: Failing over lustre-OST0000 [10328.063589] LustreError: 8961:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10328.067544] LustreError: 8961:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10328.088346] Lustre: server umount lustre-OST0000 complete [10331.351916] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10331.360486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10331.428258] Lustre: 9557:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [10331.525878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10331.532711] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10333.167041] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10333.388826] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [10333.508906] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10333.509036] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [10333.509038] Lustre: Skipped 1 previous similar message [10334.826196] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 19:18:07 (1746141487) [10340.939066] Lustre: Failing over lustre-OST0000 [10340.990131] Lustre: server umount lustre-OST0000 complete [10345.130030] LustreError: 12223:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10345.141249] LustreError: 12223:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 7 previous similar messages [10348.493308] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10348.500926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10348.564796] Lustre: 11478:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [10348.689251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10348.699248] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10349.909229] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10350.182591] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10350.187753] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.147@tcp (at 0@lo) [10350.190842] Lustre: Skipped 1 previous similar message [10351.145214] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [10353.027151] Lustre: DEBUG MARKER: == sanity test 165g: ofd_access_log_reader --keepalive works ========================================================== 19:18:25 (1746141505) [10394.608795] Lustre: Failing over lustre-OST0000 [10394.636949] LustreError: 12424:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [10394.641058] LustreError: 12424:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10394.660187] Lustre: server umount lustre-OST0000 complete [10395.270734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10395.271789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10395.271790] Lustre: Skipped 4 previous similar messages [10395.284831] LustreError: Skipped 1 previous similar message [10402.132688] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [10402.139430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10402.200231] Lustre: 13568:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [10402.276242] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10402.285474] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10403.905726] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10403.918126] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.147@tcp (at 0@lo) [10403.918302] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10403.928003] Lustre: Skipped 1 previous similar message [10404.165429] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [10405.895683] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 19:19:18 (1746141558) [10406.538154] Lustre: DEBUG MARKER: creating a 10 Mb file [10429.063818] Lustre: DEBUG MARKER: starting reads [10429.978599] Lustre: DEBUG MARKER: truncating the file [10431.011101] Lustre: DEBUG MARKER: killing dd [10431.794877] Lustre: DEBUG MARKER: removing the temporary file [10434.624243] Lustre: DEBUG MARKER: == sanity test 170a: test lctl df to handle corrupted log ========================================================== 19:19:46 (1746141586) [10438.798722] Lustre: DEBUG MARKER: == sanity test 170b: check filename encoding ============= 19:19:50 (1746141590) [10451.334764] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 19:20:03 (1746141603) [10457.240535] Lustre: DEBUG MARKER: == sanity test 172: manual device removal with lctl cleanup/detach ================================================================ 19:20:09 (1746141609) [10460.843610] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 19:20:13 (1746141613) [10461.398456] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [10462.065722] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 19:20:14 (1746141614) [10462.931900] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_module obdecho/obdecho [10469.927979] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 19:20:22 (1746141622) [10470.761217] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing load_module obdecho/obdecho [10470.819020] Lustre: Echo OBD driver; http://www.lustre.org/ [10478.495460] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 19:20:30 (1746141630) [10524.983653] Lustre: DEBUG MARKER: == sanity test 182a: Test parallel modify metadata operations from mdc ========================================================== 19:21:17 (1746141677) [10546.855904] Lustre: DEBUG MARKER: == sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 19:21:38 (1746141698) [10912.117287] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 19:27:44 (1746142064) [10912.840244] Lustre: *** cfs_fail_loc=148, val=0*** [10916.954607] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 19:27:48 (1746142068) [10921.520865] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 19:27:53 (1746142073) [10924.880264] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 19:27:57 (1746142077) [10946.025214] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 19:28:18 (1746142098) [10951.537245] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 19:28:23 (1746142103) [10956.917874] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 19:28:28 (1746142108) [10961.213106] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 19:28:33 (1746142113) [10965.760698] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 19:28:37 (1746142117) [10973.259121] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 19:28:45 (1746142125) [10976.932598] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 19:28:49 (1746142129) [10979.647377] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 19:28:51 (1746142131) [10999.843735] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 19:29:11 (1746142151) [11003.321662] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 19:29:15 (1746142155) [11006.912379] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 19:29:18 (1746142158) [11010.301372] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 19:29:22 (1746142162) [11013.642508] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 19:29:25 (1746142165) [11017.315241] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 19:29:29 (1746142169) [11020.423432] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 19:29:32 (1746142172) [11023.791089] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 19:29:35 (1746142175) [11027.831154] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 19:29:39 (1746142179) [11032.317451] Lustre: lustre-MDD0000: changelog on [11032.319815] Lustre: Skipped 5 previous similar messages [11035.745145] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [11036.684024] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.32360 [11038.044077] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [11038.964115] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.28684 [11040.542491] Lustre: DEBUG MARKER: Test: lfs mkdir -i 1 /mnt/lustre/d205a.sanity.remote [11041.384549] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.22622 [11042.674438] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [11043.647890] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.13428 [11044.807005] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [11045.530993] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.23882 [11046.624982] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [11047.337059] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.18808 [11048.523039] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11049.218307] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.32646 [11051.002068] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [11051.863185] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.22079 [11053.606330] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [11054.613191] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.27158 [11055.911363] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [11056.725854] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.28760 [11058.360771] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [11059.130458] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.12724 [11060.340031] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [11061.104847] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.12104 [11065.937914] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11066.713341] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg247-client.v [11068.449648] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11069.610457] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg247-client.E [11071.064481] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [11071.865983] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.oleg247-client.v [11076.098181] Lustre: lustre-MDD0001: changelog off [11076.100956] Lustre: Skipped 6 previous similar messages [11081.698986] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 19:30:33 (1746142233) [11087.728171] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 19:30:39 (1746142239) [11091.466393] Lustre: DEBUG MARKER: == sanity test 205d: verify the format of some stats files ========================================================== 19:30:43 (1746142243) [11100.228368] Lustre: DEBUG MARKER: == sanity test 205e: verify the output of lljobstat ====== 19:30:52 (1746142252) [11110.590463] Lustre: DEBUG MARKER: == sanity test 205f: verify qos_ost_weights YAML format == 19:31:02 (1746142262) [11115.636920] Lustre: DEBUG MARKER: == sanity test 205g: stress test for job_stats procfile == 19:31:07 (1746142267) [11210.978163] Lustre: DEBUG MARKER: == sanity test 205h: check jobid xattr is stored correctly ========================================================== 19:32:43 (1746142363) [11218.229929] Lustre: DEBUG MARKER: == sanity test 205i: check job_xattr parameter accepts and rejects values correctly ========================================================== 19:32:50 (1746142370) [11226.841910] Lustre: DEBUG MARKER: == sanity test 205k: Verify '?' operator on job stats ==== 19:32:58 (1746142378) [11232.732440] Lustre: DEBUG MARKER: == sanity test 205l: Verify job stats can scale ========== 19:33:04 (1746142384) [11311.340316] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 19:34:23 (1746142463) [11314.526197] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 19:34:26 (1746142466) [11317.799273] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 19:34:29 (1746142469) [11320.918380] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 19:34:33 (1746142473) [11327.939597] Lustre: Failing over lustre-MDT0000 [11328.069716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11328.077310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11328.090247] Lustre: Skipped 1 previous similar message [11328.092051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11328.120071] LustreError: 25600:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11328.123387] LustreError: 25600:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11328.217433] Lustre: server umount lustre-MDT0000 complete [11331.573227] LustreError: 12996:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11331.582293] LustreError: 12996:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [11341.348445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11341.413728] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11341.563214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11341.588976] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11342.848940] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [11343.392463] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11346.569030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [11346.574481] Lustre: Skipped 1 previous similar message [11346.601328] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11346.632759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:36217 to 0x280000403:36257) [11346.634553] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:36754 to 0x2c0000403:36769) [11348.077386] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11348.815853] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11353.062526] Lustre: Failing over lustre-MDT0000 [11353.136634] LustreError: 26887:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11353.139713] LustreError: 26887:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11353.188478] Lustre: server umount lustre-MDT0000 complete [11356.583562] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11356.585052] LustreError: 1764:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11356.585054] LustreError: 1764:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 9 previous similar messages [11356.605181] Lustre: Skipped 6 previous similar messages [11366.672071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11366.764302] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11366.933884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11366.971019] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11367.632854] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11368.593020] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [11371.935483] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [11371.939145] Lustre: Skipped 3 previous similar messages [11371.954473] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11371.992647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:36217 to 0x280000403:36289) [11371.992989] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:36754 to 0x2c0000403:36801) [11373.525483] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11374.215096] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11379.418597] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 19:35:31 (1746142531) [11393.137585] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 19:35:45 (1746142545) [11399.109530] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 19:35:51 (1746142551) [11403.142387] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 19:35:55 (1746142555) [11416.305643] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 19:36:08 (1746142568) [11434.958351] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 19:36:27 (1746142587) [11438.301768] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 19:36:30 (1746142590) [11448.140484] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with embedded hyphen ('-') ========================================================== 19:36:40 (1746142600) [11452.999736] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 19:36:45 (1746142605) [11453.819537] Lustre: DEBUG MARKER: creating a 10 Mb file [11485.131066] Lustre: DEBUG MARKER: starting reads [11485.874448] Lustre: DEBUG MARKER: truncating the file [11486.869228] Lustre: DEBUG MARKER: killing dd [11487.604117] Lustre: DEBUG MARKER: removing the temporary file [11490.507144] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 19:37:22 (1746142642) [11493.691133] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 19:37:25 (1746142645) [11495.864769] Lustre: *** cfs_fail_loc=229, val=0*** [11497.253281] Lustre: *** cfs_fail_loc=229, val=1*** [11497.255007] Lustre: Skipped 2 previous similar messages [11500.793242] Lustre: *** cfs_fail_loc=229, val=0*** [11511.133383] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 19:37:43 (1746142663) [11515.785414] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 19:37:48 (1746142668) [11519.259834] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 19:37:51 (1746142671) [11523.163548] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 19:37:55 (1746142675) [11526.999980] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 19:37:59 (1746142679) [11527.143874] Lustre: lustre-OST0000: Client a02b0b9f-856d-43e8-8cd0-98d3e3f6f813 (at 192.168.202.47@tcp) reconnecting [11531.664242] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 19:38:03 (1746142683) [11534.696892] LustreError: 24888:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 sleeping for 3000ms [11537.706784] LustreError: 24888:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 awake [11542.802724] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 19:38:14 (1746142694) [11550.790633] Lustre: *** cfs_fail_loc=520, val=57344*** [11550.821239] LNet: *** cfs_fail_loc=e000, val=2147483648*** [11555.809391] Lustre: lustre-OST0000: Client a02b0b9f-856d-43e8-8cd0-98d3e3f6f813 (at 192.168.202.47@tcp) reconnecting [11565.354092] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 19:38:37 (1746142717) [11567.952070] LustreError: 25019:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 sleeping for 22000ms [11587.894307] Lustre: lustre-OST0000: Client a02b0b9f-856d-43e8-8cd0-98d3e3f6f813 (at 192.168.202.47@tcp) reconnecting [11589.498240] LustreError: 25019:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout interrupted [11589.502139] LustreError: 25019:0:(ldlm_lib.c:3490:target_bulk_io()) @@@ bulk READ failed: rc = -107 req@ffff880097615500 x1830958648202624/t0(0) o3->a02b0b9f-856d-43e8-8cd0-98d3e3f6f813@192.168.202.47@tcp:0/0 lens 488/440 e 0 to 0 dl 1746142741 ref 1 fl Interpret:/200/0 rc 0/0 job:'dd.0' uid:0 gid:0 [11589.513142] Lustre: 25019:0:(service.c:2494:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/1s); client may timeout req@ffff880097615500 x1830958648202624/t0(0) o3->a02b0b9f-856d-43e8-8cd0-98d3e3f6f813@192.168.202.47@tcp:0/0 lens 488/440 e 0 to 0 dl 1746142741 ref 1 fl Complete:/200/0 rc 0/0 job:'dd.0' uid:0 gid:0 [11594.190454] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [11595.320478] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [11596.398001] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 19:39:08 (1746142748) [11600.178316] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 19:39:12 (1746142752) [11604.202408] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 19:39:16 (1746142756) [11609.522108] Lustre: DEBUG MARKER: == sanity test 226d: verify fid2path with -n and -fn option ========================================================== 19:39:21 (1746142761) [11613.749892] Lustre: DEBUG MARKER: == sanity test 226e: Verify path2fid -0 option with newline and space ========================================================== 19:39:25 (1746142765) [11617.021247] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 19:39:29 (1746142769) [11620.802009] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 19:39:32 (1746142772) [11767.928228] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 19:41:59 (1746142919) [11892.836750] Lustre: Failing over lustre-MDT0000 [11892.857363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11892.867644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11892.880530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11893.091157] LustreError: 8535:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11893.094610] LustreError: 8535:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11893.170689] Lustre: server umount lustre-MDT0000 complete [11895.596127] LustreError: 21645:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11895.603372] LustreError: 21645:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 13 previous similar messages [11898.404895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11898.511532] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11898.702510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11898.746958] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11900.582522] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11900.639297] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [11903.708455] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [11903.717144] Lustre: Skipped 3 previous similar messages [11903.737032] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11903.765586] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:42980 to 0x2c0000403:43009) [11903.766086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:42503 to 0x280000403:42529) [11932.809165] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 19:44:44 (1746143084) [12207.574627] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 19:49:19 (1746143359) [12212.032478] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 19:49:24 (1746143364) [12217.024325] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 19:49:28 (1746143368) [12262.346856] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 19:50:13 (1746143413) [12264.316429] Lustre: *** cfs_fail_loc=1801, val=0*** [12264.318519] LustreError: 12833:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240001b77:0x2f6e:0x0]/f1 failed: rc = -5 [12265.480302] LustreError: 12833:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0000: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush migrate_dir' to finish migration: rc = -1 [12265.492803] LustreError: 12833:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200004281:0x56fa:0x0]/migrate_dir failed: rc = -1 [12268.136875] Lustre: 12833:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0001: [0x240001b71:0x16:0x0]/foo is open, migrate only dentry [12275.263995] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [12276.467623] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 19:50:28 (1746143428) [12282.213568] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 19:50:34 (1746143434) [12288.573925] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 19:50:40 (1746143440) [12292.630589] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 19:50:44 (1746143444) [12297.074478] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 19:50:49 (1746143449) [12301.398943] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 19:50:53 (1746143453) [12305.708192] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 19:50:57 (1746143457) [12306.563012] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [12307.779075] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 19:50:59 (1746143459) [12380.506779] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 19:52:12 (1746143532) [12388.176097] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 19:52:19 (1746143539) [12393.330133] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 19:52:25 (1746143545) [12413.838751] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 19:52:45 (1746143565) [12441.587057] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 19:53:13 (1746143593) [12473.630643] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 19:53:45 (1746143625) [12479.879224] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 19:53:51 (1746143631) [12486.469085] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 19:53:58 (1746143638) [12493.013984] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 19:54:04 (1746143644) [12494.042236] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [12495.183402] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 19:54:07 (1746143647) [12496.049013] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [12497.428778] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 19:54:09 (1746143649) [12504.245483] Lustre: DEBUG MARKER: == sanity test 230x: dir migration check space =========== 19:54:16 (1746143656) [12513.542720] LustreError: 1299:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000007:0x1:0x0]/d230x.sanity failed: rc = -28 [12513.552185] LustreError: 1299:0:(mdt_reint.c:2523:mdt_reint_migrate()) Skipped 1 previous similar message [12549.796860] Lustre: DEBUG MARKER: == sanity test 230y: unlink dir with bad hash type ======= 19:55:01 (1746143701) [12552.562603] Lustre: *** cfs_fail_loc=1802, val=0*** [12552.671101] LustreError: 1299:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240001b77:0x359c:0x0]/d11 failed: rc = -77 [12562.289346] Lustre: DEBUG MARKER: == sanity test 230z: resume dir migration with bad hash type ========================================================== 19:55:14 (1746143714) [12564.830032] Lustre: *** cfs_fail_loc=1802, val=0*** [12564.831742] Lustre: Skipped 1 previous similar message [12564.937571] LustreError: 1299:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240001b77:0x35d4:0x0]/d95 failed: rc = -77 [12599.340300] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 19:55:51 (1746143751) [12605.620446] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 19:55:57 (1746143757) [12637.165382] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 19:56:29 (1746143789) [12637.727556] Lustre: *** cfs_fail_loc=31c, val=0*** [12640.238733] Lustre: Failing over lustre-OST0000 [12640.361539] LustreError: 22062:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [12640.364745] LustreError: 22062:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [12640.440949] Lustre: server umount lustre-OST0000 complete [12643.001990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12643.008553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12643.014028] Lustre: Skipped 3 previous similar messages [12643.018687] LustreError: 12742:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12643.026060] LustreError: 12742:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [12644.631390] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [12644.639176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12644.696900] Lustre: 22656:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [12644.807114] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12644.815583] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [12646.522706] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [12646.810456] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [12646.811252] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [12646.811253] Lustre: Skipped 3 previous similar messages [12647.129711] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [12651.416805] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 19:56:43 (1746143803) [12652.152887] Lustre: *** cfs_fail_loc=31c, val=0*** [12652.154614] LustreError: 21386:0:(ldlm_request.c:475:ldlm_cli_enqueue_local()) ### delayed lvb init failed (rc -12) ns: filter-lustre-OST0000_UUID lock: ffff88006131d000/0x37546a12eee6d79b lrc: 2/0,0 mode: --/PR res: [0x280000403:0xd349:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 21386 timeout: 0 lvb_type: 0 [12654.035024] Lustre: Failing over lustre-OST0000 [12654.063345] LustreError: 23703:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [12654.067414] LustreError: 23703:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12654.098949] Lustre: server umount lustre-OST0000 complete [12654.824195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12654.830424] Lustre: Skipped 1 previous similar message [12654.834361] LustreError: 3060:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12654.843877] LustreError: 3060:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [12657.955546] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [12657.961106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12658.017117] Lustre: 24298:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [12658.128893] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12658.138626] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [12658.160628] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [12660.062862] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.147@tcp (at 0@lo) [12660.064237] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [12660.075560] Lustre: Skipped 1 previous similar message [12660.223370] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [12664.956836] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 19:56:56 (1746143816) [12668.420308] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 19:57:00 (1746143820) [12671.293580] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 19:57:03 (1746143823) [12674.282272] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 19:57:06 (1746143826) [12679.390111] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 19:57:11 (1746143831) [12683.676326] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 19:57:15 (1746143835) [12687.740896] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 19:57:19 (1746143839) [12748.284493] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 19:58:20 (1746143900) [12749.712697] Lustre: *** cfs_fail_loc=2100, val=0*** [12758.607075] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 19:58:30 (1746143910) [12766.286970] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 19:58:38 (1746143918) [12767.725177] Lustre: *** cfs_fail_loc=713, val=1*** [12767.726981] Lustre: Skipped 1 previous similar message [12774.075133] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 19:58:46 (1746143926) [12825.736100] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 19:59:37 (1746143977) [12847.789637] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 19:59:59 (1746143999) [12848.469420] Lustre: *** cfs_fail_loc=105, val=0*** [12851.708785] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 20:00:03 (1746144003) [13060.499406] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 20:03:32 (1746144212) [13127.972990] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 20:04:40 (1746144280) [13131.991687] Lustre: DEBUG MARKER: == sanity test 245a: check mdc connection flag/data: multiple modify RPCs ========================================================== 20:04:44 (1746144284) [13135.317103] Lustre: DEBUG MARKER: == sanity test 245b: check osp connection flag/data: multiple modify RPCs ========================================================== 20:04:47 (1746144287) [13140.174920] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 20:04:51 (1746144291) [13145.088651] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 20:04:57 (1746144297) [13149.573160] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 20:05:01 (1746144301) [13155.735509] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 20:05:07 (1746144307) [13161.467171] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 20:05:13 (1746144313) [13165.794843] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 20:05:17 (1746144317) [13167.247002] Lustre: enable_remote_subdir_mount is deprecated, it's always enabled. [13175.491093] Lustre: DEBUG MARKER: == sanity test 247g: striped directory submount revalidate ROOT from cache ========================================================== 20:05:27 (1746144327) [13180.999499] Lustre: DEBUG MARKER: == sanity test 247h: remote directory submount revalidate ROOT from cache ========================================================== 20:05:33 (1746144333) [13188.597634] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 20:05:40 (1746144340) [13251.617344] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 20:06:43 (1746144403) [13281.474422] Lustre: DEBUG MARKER: == sanity test 248c: verify whole file read behavior ===== 20:07:13 (1746144433) [13296.130394] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 20:07:27 (1746144447) [13299.801653] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 20:07:31 (1746144451) [13303.245145] Lustre: DEBUG MARKER: == sanity test 251a: Handling short read and write correctly ========================================================== 20:07:35 (1746144455) [13307.641404] Lustre: DEBUG MARKER: == sanity test 251b: short read restore offset correctly ========================================================== 20:07:39 (1746144459) [13315.845805] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 20:07:48 (1746144468) [13320.768614] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 20:07:53 (1746144473) [13353.483400] LustreError: 2782:0:(lod_qos.c:1380:lod_ost_alloc_specific()) can't lstripe objid [0x200004284:0x41:0x0]: have 0 want 1 [13394.131576] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 20:09:06 (1746144546) [13395.737830] Lustre: lustre-MDD0000: changelog on [13395.739303] Lustre: Skipped 1 previous similar message [13401.055254] Lustre: lustre-MDD0001: changelog off [13401.058014] Lustre: Skipped 1 previous similar message [13403.899604] Lustre: DEBUG MARKER: SKIP: sanity test_255a skipping excluded test 255a (base 255) [13404.805798] Lustre: DEBUG MARKER: SKIP: sanity test_255b skipping excluded test 255b (base 255) [13406.019440] Lustre: DEBUG MARKER: SKIP: sanity test_255c skipping excluded test 255c (base 255) [13407.087105] Lustre: DEBUG MARKER: SKIP: sanity test_256 skipping excluded test 256 [13407.848748] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 20:09:20 (1746144560) [13408.629303] Lustre: *** cfs_fail_loc=161, val=0*** [13409.360345] Lustre: Failing over lustre-MDT0001 [13409.366954] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13409.372135] Lustre: Skipped 2 previous similar messages [13409.372458] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13414.197741] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.47@tcp (stopping) [13414.201226] Lustre: Skipped 2 previous similar messages [13415.635482] LustreError: 8713:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [13415.638846] LustreError: 8713:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13415.712821] Lustre: server umount lustre-MDT0001 complete [13419.190912] LustreError: 31748:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13419.586450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13419.672871] Lustre: 9331:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [13419.886489] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13419.919389] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [13421.081175] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [13421.309116] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13423.332160] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [13424.894209] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.202.147@tcp (at 0@lo) [13424.897368] Lustre: Skipped 1 previous similar message [13424.908526] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [13424.929943] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:22666 to 0x2c0000400:22689) [13424.932310] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:22647 to 0x280000400:22689) [13432.221221] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 20:09:44 (1746144584) [13435.688445] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 20:09:47 (1746144587) [13439.100779] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 20:09:51 (1746144591) [13449.379139] Lustre: *** cfs_fail_loc=2301, val=0*** [13450.872333] Lustre: Failing over lustre-OST0000 [13450.903943] LustreError: 11692:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13450.908462] LustreError: 11692:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [13450.994880] Lustre: server umount lustre-OST0000 complete [13451.266622] LustreError: 3060:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13451.274024] LustreError: 3060:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [13454.953215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13454.963253] Lustre: Skipped 1 previous similar message [13454.968191] LustreError: 12224:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13454.975044] LustreError: 12224:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [13455.153231] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13455.173558] LDISKFS-fs (dm-2): 1 truncate cleaned up [13455.175802] LDISKFS-fs (dm-2): recovery complete [13455.187604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13455.263474] Lustre: 12365:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [13455.360038] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13455.370025] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13456.262647] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13457.188425] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13457.189296] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.147@tcp (at 0@lo) [13457.189297] Lustre: Skipped 2 previous similar messages [13457.268422] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13464.087492] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 20:10:16 (1746144616) [13466.969696] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 20:10:19 (1746144619) [13473.407460] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 20:10:25 (1746144625) [13476.636740] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 20:10:28 (1746144628) [13479.868308] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 20:10:32 (1746144632) [13483.008708] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 20:10:35 (1746144635) [13486.705633] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 20:10:38 (1746144638) [13489.351344] Lustre: Increasing provided stripe size to a minimum value 64 [13494.522836] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 20:10:46 (1746144646) [13497.247978] Lustre: *** cfs_fail_loc=168, val=40*** [13497.925340] Lustre: *** cfs_fail_loc=168, val=20*** [13497.928043] Lustre: Skipped 2 previous similar messages [13499.296879] Lustre: *** cfs_fail_loc=168, val=15*** [13499.299955] Lustre: Skipped 5 previous similar messages [13504.084693] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 20:10:56 (1746144656) [13508.873426] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 20:11:00 (1746144660) [13512.734272] Lustre: DEBUG MARKER: == sanity test 270j: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 20:11:04 (1746144664) [13516.945130] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 20:11:08 (1746144668) [13520.549229] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 20:11:12 (1746144672) [13524.434607] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 20:11:16 (1746144676) [13528.118507] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 20:11:20 (1746144680) [13603.638296] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 20:12:35 (1746144755) [13607.761504] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 20:12:39 (1746144759) [13612.622826] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 20:12:44 (1746144764) [13617.229838] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 20:12:49 (1746144769) [13620.670780] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 20:12:52 (1746144772) [13625.849231] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 20:12:58 (1746144778) [13630.412723] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 20:13:02 (1746144782) [13635.616413] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 20:13:07 (1746144787) [13640.392163] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 20:13:12 (1746144792) [13644.352776] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 20:13:16 (1746144796) [13648.273310] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 20:13:20 (1746144800) [13649.758681] LustreError: 12833:0:(mdt_io.c:571:mdt_commitrw_write()) cfs_fail_timeout id 16b sleeping for 2000ms [13651.765018] LustreError: 12833:0:(mdt_io.c:571:mdt_commitrw_write()) cfs_fail_timeout id 16b awake [13655.593248] Lustre: DEBUG MARKER: == sanity test 273c: race writeback and object destroy === 20:13:27 (1746144807) [13660.844057] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 20:13:32 (1746144812) [13666.924612] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 20:13:38 (1746144818) [13668.042443] Lustre: Failing over lustre-OST0000 [13668.045641] LustreError: 25080:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13668.106613] LustreError: 25072:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13668.113185] LustreError: 25072:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13668.189028] Lustre: server umount lustre-OST0000 complete [13668.728882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13668.734165] Lustre: Skipped 1 previous similar message [13668.738200] LustreError: 20303:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13668.745952] LustreError: 20303:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [13671.811972] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13671.821230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13671.887869] Lustre: 26321:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [13672.030609] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13672.042054] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13673.524572] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13673.996400] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13673.996566] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.147@tcp (at 0@lo) [13673.996567] Lustre: Skipped 1 previous similar message [13674.348995] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13676.648088] Lustre: Failing over lustre-OST0000 [13676.649372] LustreError: 27618:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13676.717989] LustreError: 27611:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13676.720504] LustreError: 27611:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13676.849685] Lustre: server umount lustre-OST0000 complete [13677.030767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13677.037273] Lustre: Skipped 1 previous similar message [13677.040256] LustreError: 3060:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13677.048755] LustreError: 3060:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [13680.759974] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13680.769314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13680.829943] Lustre: 28843:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [13680.944315] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13680.963943] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13681.659850] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13682.880644] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [13682.882889] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.147@tcp (at 0@lo) [13682.882890] Lustre: Skipped 1 previous similar message [13683.099514] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13684.555358] Lustre: Failing over lustre-OST0000 [13684.560957] LustreError: 30171:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13684.563880] LustreError: 30171:0:(obd_class.h:479:obd_check_dev()) Skipped 2 previous similar messages [13684.617625] LustreError: 30164:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13684.620367] LustreError: 30164:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13684.656560] Lustre: server umount lustre-OST0000 complete [13685.944279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13685.950809] Lustre: Skipped 2 previous similar messages [13688.049764] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13688.062086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13688.214767] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13689.739939] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13690.373806] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13692.095696] Lustre: Failing over lustre-OST0000 [13692.098929] LustreError: 32717:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13692.208508] Lustre: server umount lustre-OST0000 complete [13693.209845] LustreError: 3060:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13693.220475] LustreError: 3060:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [13695.981078] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13695.993145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13696.057778] Lustre: 1539:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [13696.065798] Lustre: 1539:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message [13696.190617] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13696.196358] Lustre: Skipped 1 previous similar message [13698.011548] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [13698.012128] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13698.012129] Lustre: Skipped 1 previous similar message [13698.024443] Lustre: Skipped 3 previous similar messages [13698.166154] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13699.818001] Lustre: Failing over lustre-OST0000 [13699.822022] LustreError: 2879:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13699.878225] LustreError: 2871:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13699.885499] LustreError: 2871:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [13699.914169] Lustre: server umount lustre-OST0000 complete [13703.859264] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13703.865315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13704.125083] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13704.130010] Lustre: Skipped 1 previous similar message [13706.573501] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13708.730204] Lustre: Failing over lustre-OST0000 [13708.735766] LustreError: 5404:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13708.738684] LustreError: 5404:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [13708.904665] Lustre: server umount lustre-OST0000 complete [13709.127128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13709.132704] Lustre: Skipped 4 previous similar messages [13714.582063] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13714.592921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13714.707598] Lustre: 6614:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [13714.724192] Lustre: 6614:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message [13714.893784] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13714.896885] Lustre: Skipped 1 previous similar message [13716.400864] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13716.403423] Lustre: Skipped 2 previous similar messages [13716.684678] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13716.687392] Lustre: Skipped 1 previous similar message [13716.688720] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [13716.688722] Lustre: Skipped 3 previous similar messages [13717.838620] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13719.996648] Lustre: Failing over lustre-OST0000 [13720.099905] LustreError: 7928:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13720.116705] LustreError: 7928:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [13720.433126] Lustre: server umount lustre-OST0000 complete [13721.703167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13721.706769] LustreError: Skipped 1 previous similar message [13725.127155] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13725.144217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13725.368295] LustreError: 9253:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13725.375557] LustreError: 9253:0:(obd_class.h:479:obd_check_dev()) Skipped 6 previous similar messages [13725.471604] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13725.475786] Lustre: Skipped 1 previous similar message [13728.629189] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13730.473156] Lustre: Failing over lustre-OST0000 [13730.537485] Lustre: server umount lustre-OST0000 complete [13731.736240] LustreError: 21387:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13731.755772] LustreError: 21387:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 13 previous similar messages [13734.840261] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13734.855121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13737.629294] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13740.052461] Lustre: Failing over lustre-OST0000 [13740.106503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13740.109414] Lustre: Skipped 4 previous similar messages [13740.136871] Lustre: server umount lustre-OST0000 complete [13744.491672] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13744.504591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13747.447482] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13749.040721] Lustre: Failing over lustre-OST0000 [13749.094240] Lustre: server umount lustre-OST0000 complete [13749.815169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13749.823138] Lustre: Skipped 7 previous similar messages [13752.618786] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13752.625958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13752.702227] Lustre: 16853:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [13752.707954] Lustre: 16853:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 3 previous similar messages [13752.871844] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13752.880343] Lustre: Skipped 3 previous similar messages [13754.070446] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13754.073603] Lustre: Skipped 3 previous similar messages [13754.445240] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13754.449144] Lustre: Skipped 3 previous similar messages [13754.449876] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.147@tcp (at 0@lo) [13754.449877] Lustre: Skipped 7 previous similar messages [13754.925734] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13756.649324] Lustre: Failing over lustre-OST0000 [13756.702159] LustreError: 18174:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13756.710337] LustreError: 18174:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [13756.785823] Lustre: server umount lustre-OST0000 complete [13760.998748] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13761.006771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13761.269589] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13761.275517] Lustre: Skipped 3 previous similar messages [13763.698811] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13765.776340] Lustre: Failing over lustre-OST0000 [13765.780149] LustreError: 20672:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13765.784907] LustreError: 20672:0:(obd_class.h:479:obd_check_dev()) Skipped 4 previous similar messages [13765.904627] Lustre: server umount lustre-OST0000 complete [13770.147228] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13770.154348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13773.235740] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13775.257481] Lustre: Failing over lustre-OST0000 [13775.338025] Lustre: server umount lustre-OST0000 complete [13779.949874] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13779.962395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13782.882173] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13784.624020] Lustre: Failing over lustre-OST0000 [13784.716319] Lustre: server umount lustre-OST0000 complete [13788.929165] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13788.935470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13791.469972] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13793.231570] Lustre: Failing over lustre-OST0000 [13793.303112] Lustre: server umount lustre-OST0000 complete [13795.846874] LustreError: 3065:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13795.855580] LustreError: 3065:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 18 previous similar messages [13797.085001] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13797.091352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13799.514441] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13801.223967] Lustre: Failing over lustre-OST0000 [13801.291769] Lustre: server umount lustre-OST0000 complete [13805.308351] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13805.313503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13808.146939] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13810.215904] Lustre: Failing over lustre-OST0000 [13810.285734] Lustre: server umount lustre-OST0000 complete [13814.320240] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13814.330050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13816.997786] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13818.883706] Lustre: Failing over lustre-OST0000 [13819.028285] Lustre: server umount lustre-OST0000 complete [13819.541975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13819.549062] Lustre: Skipped 16 previous similar messages [13823.500762] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13823.513606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13823.604336] Lustre: 4756:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.147@tcp: no remote llog for lustre-sptlrpc, check MGS config [13823.609702] Lustre: 4756:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 7 previous similar messages [13823.758967] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13823.763508] Lustre: Skipped 7 previous similar messages [13825.225355] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13825.229556] Lustre: Skipped 7 previous similar messages [13825.577000] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13825.583951] Lustre: Skipped 7 previous similar messages [13825.586085] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.147@tcp (at 0@lo) [13825.586087] Lustre: Skipped 15 previous similar messages [13826.127280] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13827.954691] Lustre: Failing over lustre-OST0000 [13827.997748] LustreError: 6068:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13828.000645] LustreError: 6068:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [13828.024153] Lustre: server umount lustre-OST0000 complete [13832.375639] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13832.382861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13832.526433] LustreError: 7316:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13832.533244] LustreError: 7316:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [13832.682895] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13832.686662] Lustre: Skipped 7 previous similar messages [13835.324381] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13837.055767] Lustre: Failing over lustre-OST0000 [13837.161069] Lustre: server umount lustre-OST0000 complete [13841.071336] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13841.079100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13843.901094] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13849.206416] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 20:16:41 (1746145001) [13852.918821] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 20:16:44 (1746145004) [13854.276993] Lustre: Failing over lustre-MDT0001 [13854.289768] Lustre: Failing over lustre-MDT0000 [13854.395348] LustreError: 11841:0:(tgt_mount.c:1756:server_put_super()) cfs_race id 60c sleeping [13855.936118] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.47@tcp (stopping) [13859.401968] LustreError: 11841:0:(tgt_mount.c:1756:server_put_super()) cfs_fail_race id 60c awake: rc=0 [13859.433614] Lustre: server umount lustre-MDT0001 complete [13860.938153] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.47@tcp (stopping) [13860.947491] Lustre: Skipped 2 previous similar messages [13870.959023] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.47@tcp (stopping) [13870.966115] Lustre: Skipped 5 previous similar messages [13884.811435] Lustre: server umount lustre-MDT0000 complete [13889.307643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13889.384703] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13889.483243] LustreError: 12759:0:(tgt_mount.c:1486:server_start_targets()) cfs_fail_race id 60c waking [13891.657158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13893.464371] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13899.204893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13899.583694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13901.283394] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13903.585534] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [13911.115461] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:22695 to 0x2c0000400:22721) [13911.115505] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:22706 to 0x280000400:22721) [13911.142255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:54133 to 0x280000403:54177) [13911.142328] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:54606 to 0x2c0000403:54625) [13917.231391] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 20:17:49 (1746145069) [13918.892117] LustreError: 12766:0:(llog_osd.c:1480:llog_osd_open()) cfs_race id 15e sleeping [13920.743700] Lustre: Failing over lustre-MDT0000 [13920.823717] LustreError: 14911:0:(tgt_mount.c:128:server_deregister_mount()) cfs_fail_race id 15e waking [13920.831492] LustreError: 12766:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e awake: rc=3067 [13921.158422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13921.842884] LustreError: 14911:0:(tgt_mount.c:128:server_deregister_mount()) cfs_fail_race id 15e waking [13921.870294] Lustre: server umount lustre-MDT0000 complete [13926.614515] LustreError: 12769:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13926.623294] LustreError: 12769:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 41 previous similar messages [13926.970306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13927.079968] LustreError: MGC192.168.202.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13927.114595] LustreError: 15517:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13928.213353] LustreError: 15516:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13929.221002] LustreError: 15516:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13931.237120] LustreError: 15516:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13931.240689] LustreError: 15516:0:(llog_osd.c:1480:llog_osd_open()) Skipped 1 previous similar message [13935.284629] LustreError: 15516:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13935.290096] LustreError: 15516:0:(llog_osd.c:1480:llog_osd_open()) Skipped 3 previous similar messages [13939.314216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13940.319447] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:54606 to 0x2c0000403:54657) [13940.321752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:54133 to 0x280000403:54209) [13940.603991] Lustre: DEBUG MARKER: oleg247-server.virtnet: executing set_default_debug all all [13943.668903] LustreError: 15515:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [13943.671364] LustreError: 15515:0:(llog_osd.c:1480:llog_osd_open()) Skipped 5 previous similar messages [13952.002477] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 20:18:24 (1746145104) [13956.707050] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 20:18:28 (1746145108) [13980.951200] Lustre: DEBUG MARKER: == sanity test 300c: chown [14158.625979] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 20:21:50 (1746145310) [14163.623471] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ========================================================== 20:21:55 (1746145315) [14167.151424] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ========================================================== 20:21:59 (1746145319) [14170.940894] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ========================================================== 20:22:02 (1746145322) [14178.233496] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory ========================================================== 20:22:10 (1746145330) [14185.106611] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory ========================================================== 20:22:17 (1746145337) [14221.048014] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ============ 20:22:53 (1746145373) [14224.473333] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ======== 20:22:56 (1746145376) [14227.935363] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ========================================================== 20:23:00 (1746145380) [14231.899797] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS ========================================================== 20:23:04 (1746145384) [14232.580864] Lustre: DEBUG MARKER: SKIP: sanity test_300m Only for single MDT [14233.475792] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA ========================================================== 20:23:05 (1746145385) [14241.211087] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [14241.934455] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================================== 20:23:14 (1746145394) [14242.525809] Lustre: *** cfs_fail_loc=1704, val=0*** [14242.530840] Lustre: 12233:0:(out_handler.c:911:out_tx_end()) lustre-MDT0001-osd: error during execution of #8 from /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:670: rc = -28 [14242.536552] LustreError: 12233:0:(out_handler.c:921:out_tx_end()) lustre-MDT0001-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:425: rc = -524 [14245.050223] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ========================================================== 20:23:17 (1746145397) [14247.598210] Lustre: DEBUG MARKER: == sanity test 300r: test -1 striped directory =========== 20:23:19 (1746145399) [14250.700550] Lustre: DEBUG MARKER: == sanity test 300s: test lfs mkdir -c without -i ======== 20:23:22 (1746145402) [14253.958890] Lustre: DEBUG MARKER: == sanity test 300t: test max_mdt_stripecount ============ 20:23:26 (1746145406) [14259.914492] Lustre: DEBUG MARKER: == sanity test 300ua: basic overstriped dir sanity test == 20:23:31 (1746145411) [14265.337378] Lustre: DEBUG MARKER: == sanity test 300ub: test MDT overstriping interface [14269.253127] Lustre: DEBUG MARKER: == sanity test 300uc: test MDT overstriping as default [14272.731810] Lustre: DEBUG MARKER: == sanity test 300ud: dir split ========================== 20:23:44 (1746145424) [14366.049857] Lustre: DEBUG MARKER: == sanity test 300ue: dir merge ========================== 20:25:18 (1746145518) [14431.336468] Lustre: DEBUG MARKER: == sanity test 300uf: migrate with too many local locks == 20:26:23 (1746145583) [14436.171811] Lustre: DEBUG MARKER: == sanity test 300ug: migrate overstriped dirs =========== 20:26:28 (1746145588) [14436.956777] Lustre: 17440:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 3793 > trans_max 3200 [14436.960780] Lustre: 17440:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 6/24/0, destroy: 6/24/0 [14436.966302] Lustre: 17440:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 31/2640/0 [14436.971087] Lustre: 17440:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 0/0/0 [14436.976632] Lustre: 17440:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 47/938/0, delete: 32/50/0 [14436.980570] Lustre: 17440:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 30/30/0, ref_del: 28/28/0 [14436.986328] CPU: 3 PID: 17440 Comm: mdt_io00_003 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [14436.992775] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [14436.996571] Call Trace: [14436.997432] [] dump_stack+0x19/0x1b [14436.999310] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [14437.001526] [] ? __llog_ctxt_put+0xd3/0x140 [obdclass] [14437.003793] [] top_trans_start+0x351/0xab0 [ptlrpc] [14437.006163] [] lod_trans_start+0x7f/0x2e0 [lod] [14437.011054] [] mdd_trans_start+0x14/0x20 [mdd] [14437.015243] [] mdd_migrate_object+0x12ee/0x2470 [mdd] [14437.019488] [] ? _raw_spin_unlock+0xe/0x20 [14437.024761] [] ? __mdt_stripe_get+0xe1/0x4e0 [mdt] [14437.034814] [] mdd_migrate+0x28/0x30 [mdd] [14437.037325] [] mdt_reint_migrate+0x1b9f/0x24b0 [mdt] [14437.039836] [] mdt_reint_rec+0x87/0x240 [mdt] [14437.041705] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [14437.044035] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [14437.046125] [] mdt_reint+0x67/0x150 [mdt] [14437.048118] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [14437.050372] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [14437.052703] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [14437.054791] [] ? put_prev_entity+0x31/0x400 [14437.056761] [] ? do_raw_spin_unlock+0x49/0x90 [14437.062032] [] ? _raw_spin_unlock_irq+0xe/0x30 [14437.066012] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [14437.073982] [] kthread+0xe4/0xf0 [14437.078775] [] ? kthread_create_on_node+0x140/0x140 [14437.083143] [] ret_from_fork_nospec_begin+0x7/0x21 [14437.087129] [] ? kthread_create_on_node+0x140/0x140 [14441.687286] Lustre: DEBUG MARKER: == sanity test 300uh: overstripe tunable max_stripes_per_mdt ========================================================== 20:26:33 (1746145593) [14449.100547] Lustre: DEBUG MARKER: == sanity test 300ui: overstripe is not supported on one MDT system ========================================================== 20:26:40 (1746145600) [14450.030702] Lustre: DEBUG MARKER: SKIP: sanity test_300ui 1 MDT only [14450.841911] Lustre: DEBUG MARKER: == sanity test 300uj: overstriped dir with -C -N sanity test ========================================================== 20:26:43 (1746145603) [14455.376094] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ============= 20:26:47 (1746145607) [14459.642133] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open ========================================================== 20:26:51 (1746145611) [14464.215230] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ========================================================== 20:26:56 (1746145616) [14465.098235] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [14466.050543] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ========================================================== 20:26:58 (1746145618) [14495.619851] Lustre: DEBUG MARKER: == sanity test 312: make sure ZFS adjusts its block size by write pattern ========================================================== 20:27:27 (1746145647) [14496.535474] Lustre: DEBUG MARKER: SKIP: sanity test_312 the test only applies to zfs [14497.599139] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ========================================================== 20:27:29 (1746145649) [14498.211106] Lustre: *** cfs_fail_loc=720, val=0*** [14498.213065] LustreError: 30381:0:(osd_handler.c:2213:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [14498.216875] LustreError: 30381:0:(ofd_objects.c:1014:ofd_object_punch()) lustre-OST0000: failed to stop transaction: rc = -5 [14502.342460] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ========================================================== 20:27:34 (1746145654) [14502.864317] Lustre: *** cfs_fail_loc=720, val=0*** [14502.870749] LustreError: 12224:0:(osd_handler.c:2213:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [14502.882179] LustreError: 21392:0:(ofd_objects.c:1081:ofd_destroy()) lustre-OST0000 failed to stop transaction: -5 [14502.882263] LustreError: 21392:0:(ofd_dev.c:1765:ofd_destroy_hdl()) lustre-OST0000: error destroying object [0x280000403:0xd8f4:0x0]: -5 [14502.900655] LustreError: 12224:0:(osd_handler.c:2213:osd_trans_stop()) Skipped 2 previous similar messages [14505.282064] LustreError: 15526:0:(osp_sync.c:1155:osp_sync_process_committed()) lustre-OST0000-osc-MDT0000: can't cancel record: rc = -2 [14509.292118] LustreError: 15528:0:(osp_sync.c:1155:osp_sync_process_committed()) lustre-OST0001-osc-MDT0000: can't cancel record: rc = -2 [14509.300311] LustreError: 15528:0:(osp_sync.c:1155:osp_sync_process_committed()) Skipped 1 previous similar message [14515.114595] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ============= 20:27:47 (1746145667) [14523.591727] Lustre: DEBUG MARKER: == sanity test 316: lfs migrate of file with large_xattr enabled ========================================================== 20:27:55 (1746145675) [14528.111383] Lustre: DEBUG MARKER: == sanity test 317: Verify blocks get correctly update after truncate ========================================================== 20:28:00 (1746145680) [14532.818195] Lustre: DEBUG MARKER: == sanity test 318: Verify async readahead tunables ====== 20:28:04 (1746145684) [14536.490752] Lustre: DEBUG MARKER: == sanity test 319: lost lease lock on migrate error ===== 20:28:08 (1746145688) [14541.783054] Lustre: 12783:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0000: [0x200005224:0x142:0x0]/xxx is open, migrate only dentry [14545.805680] Lustre: DEBUG MARKER: == sanity test 350: force NID mismatch path to be exercised ========================================================== 20:28:17 (1746145697) [14546.595819] LNet: *** cfs_fail_loc=e001, val=100*** [14546.597311] LNet: 12773:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-0@lo matched 1830958590593792 with NID mismatch 0@lo accepted (same peer ffff880135e72e00) [14547.098566] LNet: *** cfs_fail_loc=e001, val=100*** [14547.100648] LNet: Skipped 40 previous similar messages [14547.102743] LNet: 8215:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-192.168.202.47@tcp matched 1830958590599296 with NID mismatch 192.168.202.47@tcp accepted (same peer ffff8800a071f400) [14547.111446] LNet: 8215:0:(lib-ptl.c:160:lnet_try_match_md()) Skipped 40 previous similar messages [14729.814021] Lustre: DEBUG MARKER: == sanity test 360: ldiskfs unlink in a separate thread == 20:31:21 (1746145881) [14746.145593] Lustre: DEBUG MARKER: == sanity test 398a: direct IO should cancel lock otherwise lockless ========================================================== 20:31:38 (1746145898) [14750.136585] Lustre: DEBUG MARKER: == sanity test 398b: DIO and buffer IO race ============== 20:31:42 (1746145902) [14838.805178] Lustre: DEBUG MARKER: == sanity test 398c: run fio to test AIO ================= 20:33:10 (1746145990) [14871.382984] Lustre: DEBUG MARKER: == sanity test 398d: run aiocp to verify block size > stripe size ========================================================== 20:33:43 (1746146023) [14887.595304] Lustre: DEBUG MARKER: == sanity test 398e: O_Direct open cleared by fcntl doesn't cause hang ========================================================== 20:33:59 (1746146039) [14890.611592] Lustre: DEBUG MARKER: == sanity test 398f: verify aio handles ll_direct_rw_pages errors correctly ========================================================== 20:34:02 (1746146042) [14896.806908] Lustre: DEBUG MARKER: == sanity test 398g: verify parallel dio async RPC submission ========================================================== 20:34:08 (1746146048) [14897.868728] LustreError: 30695:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14897.873197] LustreError: 30695:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [14899.873780] LustreError: 699:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14899.877642] LustreError: 699:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [14900.185742] LustreError: 2702:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14900.190924] LustreError: 2702:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 5 previous similar messages [14902.200858] LustreError: 2702:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14902.204156] LustreError: 2702:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 4 previous similar messages [14902.258059] LustreError: 2702:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14904.267777] LustreError: 2702:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14904.301652] LustreError: 2702:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14906.306171] LustreError: 2702:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14908.369252] LustreError: 2702:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [14908.372551] LustreError: 2702:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [14910.379779] LustreError: 2702:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [14910.385155] LustreError: 2702:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [14920.007925] Lustre: DEBUG MARKER: == sanity test 398h: verify correctness of read [14931.991719] Lustre: DEBUG MARKER: == sanity test 398i: verify parallel dio handles ll_direct_rw_pages errors correctly ========================================================== 20:34:44 (1746146084) [14937.179200] Lustre: DEBUG MARKER: == sanity test 398j: test parallel dio where stripe size > rpc_size ========================================================== 20:34:49 (1746146089) [14950.605462] Lustre: DEBUG MARKER: == sanity test 398k: test enospc on first stripe ========= 20:35:02 (1746146102) [14963.788858] Lustre: DEBUG MARKER: SKIP: sanity test_398k 7195976 > 600000 skipping out-of-space test on OST0 [14965.005942] Lustre: DEBUG MARKER: == sanity test 398l: test enospc on intermediate stripe/RPC ========================================================== 20:35:16 (1746146116) [14970.789057] Lustre: DEBUG MARKER: SKIP: sanity test_398l 7176064 > 600000 skipping out-of-space test on OST0 [14985.313618] Lustre: DEBUG MARKER: == sanity test 398m: test RPC failures with parallel dio ========================================================== 20:35:37 (1746146137) [14986.242497] Lustre: *** cfs_fail_loc=20e, val=1*** [14987.308421] Lustre: *** cfs_fail_loc=20e, val=1*** [14987.312625] Lustre: Skipped 10 previous similar messages [14989.326536] Lustre: *** cfs_fail_loc=20e, val=1*** [14989.330135] Lustre: Skipped 1 previous similar message [14992.364691] Lustre: *** cfs_fail_loc=20e, val=1*** [14992.370709] Lustre: Skipped 5 previous similar messages [14996.400184] Lustre: *** cfs_fail_loc=20e, val=1*** [14996.402257] Lustre: Skipped 1 previous similar message [15007.428773] Lustre: *** cfs_fail_loc=20e, val=1*** [15007.430675] Lustre: Skipped 8 previous similar messages [15031.493720] Lustre: *** cfs_fail_loc=20e, val=1*** [15031.497920] Lustre: Skipped 11 previous similar messages [15064.390887] Lustre: *** cfs_fail_loc=20f, val=1*** [15064.396816] Lustre: Skipped 35 previous similar messages [15135.583488] Lustre: *** cfs_fail_loc=20e, val=2*** [15135.588701] Lustre: Skipped 55 previous similar messages [15216.142425] Lustre: DEBUG MARKER: == sanity test 398n: test append with parallel DIO ======= 20:39:28 (1746146368) [15226.930084] Lustre: DEBUG MARKER: == sanity test 398o: right kms with DIO ================== 20:39:39 (1746146379) [15227.768881] Lustre: DEBUG MARKER: sanity test_398o: @@@@@@ FAIL: bad KMS