[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] mapped at [ffffffffff2003f0] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5200 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1d87 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1c23 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1c97 00090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1d27 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1d5f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 387959210 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.651574] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.655350] pid_max: default: 32768 minimum: 301 [ 0.657800] Security Framework initialized [ 0.660688] SELinux: Initializing. [ 0.662409] SELinux: Starting in permissive mode [ 0.664599] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.672349] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.676881] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.679366] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.682738] Initializing cgroup subsys memory [ 0.684768] Initializing cgroup subsys devices [ 0.686808] Initializing cgroup subsys freezer [ 0.690162] Initializing cgroup subsys net_cls [ 0.692699] Initializing cgroup subsys blkio [ 0.695399] Initializing cgroup subsys perf_event [ 0.697870] Initializing cgroup subsys hugetlb [ 0.701094] Initializing cgroup subsys pids [ 0.703745] Initializing cgroup subsys net_prio [ 0.708090] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.713779] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.716480] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.719002] tlb_flushall_shift: 6 [ 0.720838] FEATURE SPEC_CTRL Present [ 0.722839] FEATURE IBPB_SUPPORT Present [ 0.724143] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.727381] Spectre V2 : Vulnerable [ 0.728927] Speculative Store Bypass: Vulnerable [ 0.732270] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.742569] ACPI: Core revision 20130517 [ 0.747545] ACPI: All ACPI Tables successfully acquired [ 0.768571] ftrace: allocating 30294 entries in 119 pages [ 0.835155] Enabling x2apic [ 0.836537] Enabled x2apic [ 0.838253] Switched APIC routing to physical x2apic. [ 0.842024] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.845272] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.850463] TSC deadline timer enabled [ 0.850701] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.853858] ... version: 2 [ 0.855434] ... bit width: 48 [ 0.857098] ... generic registers: 4 [ 0.858767] ... value mask: 0000ffffffffffff [ 0.860418] ... max period: 00007fffffffffff [ 0.862860] ... fixed-purpose events: 3 [ 0.864638] ... event mask: 000000070000000f [ 0.868183] KVM setup paravirtual spinlock [ 0.875317] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.884310] KVM setup async PF for cpu 1 [ 0.885992] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.888873] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.896562] KVM setup async PF for cpu 2 [ 0.872467] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.897523] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.909601] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.911344] Brought up 4 CPUs [ 0.911358] KVM setup async PF for cpu 3 [ 0.911369] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.917229] smpboot: Max logical packages: 1 [ 0.919437] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.926016] devtmpfs: initialized [ 0.928174] x86/mm: Memory block size: 128MB [ 0.935044] EVM: security.selinux [ 0.936839] EVM: security.ima [ 0.938538] EVM: security.capability [ 0.944172] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.948569] NET: Registered protocol family 16 [ 0.951477] cpuidle: using governor haltpoll [ 0.954227] ACPI: bus type PCI registered [ 0.955853] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.958749] PCI: Using configuration type 1 for base access [ 0.960898] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.975869] ACPI: Added _OSI(Module Device) [ 0.978442] ACPI: Added _OSI(Processor Device) [ 0.980876] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.983709] ACPI: Added _OSI(Processor Aggregator Device) [ 0.986276] ACPI: Added _OSI(Linux-Dell-Video) [ 0.989879] ACPI: EC: Look up EC in DSDT [ 0.995051] ACPI: Interpreter enabled [ 0.997208] ACPI: (supports S0 S3 S4 S5) [ 0.999331] ACPI: Using IOAPIC for interrupt routing [ 1.002056] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 1.006351] ACPI: Enabled 2 GPEs in block 00 to 0F [ 1.017501] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 1.020535] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 1.023867] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 1.026872] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 1.033516] acpiphp: Slot [2] registered [ 1.035738] acpiphp: Slot [5] registered [ 1.038650] acpiphp: Slot [6] registered [ 1.040886] acpiphp: Slot [7] registered [ 1.043266] acpiphp: Slot [8] registered [ 1.045871] acpiphp: Slot [9] registered [ 1.048088] acpiphp: Slot [10] registered [ 1.049993] acpiphp: Slot [3] registered [ 1.051642] acpiphp: Slot [4] registered [ 1.053500] acpiphp: Slot [11] registered [ 1.055687] acpiphp: Slot [12] registered [ 1.058051] acpiphp: Slot [13] registered [ 1.059994] acpiphp: Slot [14] registered [ 1.061859] acpiphp: Slot [15] registered [ 1.064044] acpiphp: Slot [16] registered [ 1.066281] acpiphp: Slot [17] registered [ 1.068336] acpiphp: Slot [18] registered [ 1.069922] acpiphp: Slot [19] registered [ 1.071356] acpiphp: Slot [20] registered [ 1.073339] acpiphp: Slot [21] registered [ 1.075856] acpiphp: Slot [22] registered [ 1.077950] acpiphp: Slot [23] registered [ 1.079879] acpiphp: Slot [24] registered [ 1.081679] acpiphp: Slot [25] registered [ 1.083374] acpiphp: Slot [26] registered [ 1.085147] acpiphp: Slot [27] registered [ 1.087082] acpiphp: Slot [28] registered [ 1.088995] acpiphp: Slot [29] registered [ 1.090625] acpiphp: Slot [30] registered [ 1.092647] acpiphp: Slot [31] registered [ 1.094032] PCI host bridge to bus 0000:00 [ 1.095712] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 1.097903] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 1.100366] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 1.103705] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 1.109507] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 1.112921] pci_bus 0000:00: root bus resource [bus 00-ff] [ 1.115347] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 1.116413] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 1.117819] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 1.127724] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 1.131567] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 1.134986] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 1.137789] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 1.142458] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 1.145956] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 1.146719] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 1.150020] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 1.154537] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 1.157209] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 1.168174] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 1.172284] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 1.175076] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 1.180545] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 1.184727] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 1.200451] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 1.215991] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.224715] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.229390] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.242210] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 1.248257] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.250752] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.253035] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.266122] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 1.273108] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.278804] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.284328] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.300129] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 1.305448] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.309121] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.312022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.328040] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 1.340021] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.343766] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.347454] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.367208] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 1.374372] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.377523] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.380895] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.383642] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.386018] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.392440] vgaarb: loaded [ 1.394097] SCSI subsystem initialized [ 1.395896] ACPI: bus type USB registered [ 1.397907] usbcore: registered new interface driver usbfs [ 1.400522] usbcore: registered new interface driver hub [ 1.402529] usbcore: registered new device driver usb [ 1.405498] PCI: Using ACPI for IRQ routing [ 1.407716] PCI: pci_cache_line_size set to 64 bytes [ 1.407983] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.407989] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.407991] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.409054] NetLabel: Initializing [ 1.411699] NetLabel: domain hash size = 128 [ 1.414916] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.419405] NetLabel: unlabeled traffic allowed by default [ 1.422745] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.426393] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.434003] amd_nb: Cannot enumerate AMD northbridges [ 1.443933] Switched to clocksource kvm-clock [ 1.467768] pnp: PnP ACPI init [ 1.469200] ACPI: bus type PNP registered [ 1.471407] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.471509] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.471562] pnp 00:02: [dma 2] [ 1.471600] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.471689] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.471770] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.471861] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.472334] pnp: PnP ACPI: found 6 devices [ 1.474294] ACPI: bus type PNP unregistered [ 1.500366] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.500372] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.500375] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.500377] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.500379] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.500461] NET: Registered protocol family 2 [ 1.502795] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.506334] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.510204] TCP: Hash tables configured (established 32768 bind 32768) [ 1.512824] TCP: reno registered [ 1.514466] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.517451] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.520079] NET: Registered protocol family 1 [ 1.523195] RPC: Registered named UNIX socket transport module. [ 1.524975] RPC: Registered udp transport module. [ 1.526318] RPC: Registered tcp transport module. [ 1.528298] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.531361] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.533907] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.536459] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.539764] PCI: CLS 0 bytes, default 64 [ 1.539986] Unpacking initramfs... [ 3.745364] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 3.751530] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.755885] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 3.762562] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 3.767486] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 3.771380] RAPL PMU: hw unit of domain package 2^-0 Joules [ 3.774167] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 3.783837] cryptomgr_test (52) used greatest stack depth: 14480 bytes left [ 3.784964] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 3.785022] Initialise system trusted keyring [ 3.875464] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 3.878510] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 3.909838] zpool: loaded [ 3.911364] zbud: loaded [ 3.925089] VFS: Disk quotas dquot_6.6.0 [ 3.932054] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 3.943262] NFS: Registering the id_resolver key type [ 3.947863] Key type id_resolver registered [ 3.950930] Key type id_legacy registered [ 3.952556] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 3.956784] Key type big_key registered [ 3.958754] SELinux: Registering netfilter hooks [ 3.967212] cryptomgr_test (58) used greatest stack depth: 14048 bytes left [ 3.985388] cryptomgr_test (63) used greatest stack depth: 13984 bytes left [ 3.989268] NET: Registered protocol family 38 [ 3.991535] Key type asymmetric registered [ 3.993465] Asymmetric key parser 'x509' registered [ 3.998199] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 4.006435] io scheduler noop registered [ 4.008927] io scheduler deadline registered (default) [ 4.013020] io scheduler cfq registered [ 4.014463] io scheduler mq-deadline registered [ 4.016552] io scheduler kyber registered [ 4.022996] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.026120] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.032675] intel_idle: does not run on family 6 model 62 [ 4.032849] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.036804] ACPI: Power Button [PWRF] [ 4.042854] GHES: HEST is not enabled! [ 4.145146] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 4.241801] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 4.476410] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 4.615275] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 4.779035] tsc: Refined TSC clocksource calibration: 2399.952 MHz [ 4.845896] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.881289] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 4.917969] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 4.949148] Non-volatile memory driver v1.3 [ 4.955630] Linux agpgart interface v0.103 [ 4.960164] crash memory driver: version 1.1 [ 4.967647] nbd: registered device at major 43 [ 4.997030] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 4.997082] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 4.997130] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 4.997177] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 4.997246] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 5.031134] virtio_blk virtio1: [vda] 115448 512-byte logical blocks (59.1 MB/56.3 MiB) [ 5.055475] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 5.055533] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 5.055579] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 5.055615] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 5.055651] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 5.084422] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.100615] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 5.100667] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 5.100715] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 5.100762] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 5.100817] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 5.120610] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.132963] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 5.133003] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 5.133041] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 5.133078] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 5.133110] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 5.157472] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.176745] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 5.176783] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 5.176821] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 5.176853] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 5.176885] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 5.194876] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.214172] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 5.214232] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 5.214299] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 5.214335] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 5.214370] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 5.228231] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.255212] rdac: device handler registered [ 5.264713] hp_sw: device handler registered [ 5.266331] emc: device handler registered [ 5.269810] libphy: Fixed MDIO Bus: probed [ 5.285317] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 5.289474] ehci-pci: EHCI PCI platform driver [ 5.291174] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 5.294557] ohci-pci: OHCI PCI platform driver [ 5.296596] uhci_hcd: USB Universal Host Controller Interface driver [ 5.304765] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.311770] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.318036] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.321386] mousedev: PS/2 mouse device common for all mice [ 5.327958] rtc_cmos 00:05: RTC can wake from S4 [ 5.333046] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 5.338308] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.345026] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.355032] hidraw: raw HID events driver (C) Jiri Kosina [ 5.358387] usbcore: registered new interface driver usbhid [ 5.363593] usbhid: USB HID core driver [ 5.365337] drop_monitor: Initializing network drop monitor service [ 5.373092] Netfilter messages via NETLINK v0.30. [ 5.377403] TCP: cubic registered [ 5.381852] Initializing XFRM netlink socket [ 5.384412] NET: Registered protocol family 10 [ 5.391474] NET: Registered protocol family 17 [ 5.398049] Key type dns_resolver registered [ 5.409112] mce: Using 10 MCE banks [ 5.414788] Loading compiled-in X.509 certificates [ 5.420745] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 5.424755] registered taskstats version 1 [ 5.448282] modprobe (72) used greatest stack depth: 13456 bytes left [ 5.458339] Key type trusted registered [ 5.476425] Key type encrypted registered [ 5.478162] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 5.484099] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 5.488928] rtc_cmos 00:05: setting system clock to 2025-05-01 22:54:38 UTC (1746140078) [ 5.494986] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 5.498369] Write protecting the kernel read-only data: 12288k [ 5.503052] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 5.506270] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 5.533418] random: systemd: uninitialized urandom read (16 bytes read) [ 5.548374] random: systemd: uninitialized urandom read (16 bytes read) [ 5.551605] random: systemd: uninitialized urandom read (16 bytes read) [ 5.558959] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 5.567370] systemd[1]: Detected virtualization kvm. [ 5.572246] systemd[1]: Detected architecture x86-64. [ 5.575122] systemd[1]: Running in initial RAM disk. [ 5.584556] systemd[1]: No hostname configured. [ 5.587205] systemd[1]: Set hostname to . [ 5.591577] random: systemd: uninitialized urandom read (16 bytes read) [ 5.596230] systemd[1]: Initializing machine ID from random generator. [ 5.727829] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 5.732335] random: systemd: uninitialized urandom read (16 bytes read) [ 5.735276] random: systemd: uninitialized urandom read (16 bytes read) [ 5.738414] random: systemd: uninitialized urandom read (16 bytes read) [ 5.741412] random: systemd: uninitialized urandom read (16 bytes read) [ 5.751995] random: systemd: uninitialized urandom read (16 bytes read) [ 5.760362] random: systemd: uninitialized urandom read (16 bytes read) [ 5.777527] systemd[1]: Reached target Local File Systems. [ 5.787271] systemd[1]: Reached target Timers. [ 5.799279] systemd[1]: Created slice Root Slice. [ 5.808963] systemd[1]: Listening on udev Control Socket. [ 5.820565] systemd[1]: Listening on udev Kernel Socket. [ 5.828486] systemd[1]: Listening on Journal Socket. [ 5.839481] systemd[1]: Reached target Sockets. [ 5.850830] systemd[1]: Created slice System Slice. [ 5.868500] systemd[1]: Starting Journal Service... [ 5.881121] systemd[1]: Starting Setup Virtual Console... [ 5.888835] systemd[1]: Reached target Slices. [ 5.906293] systemd[1]: Starting dracut cmdline hook... [ 5.922610] systemd[1]: Starting Load Kernel Modules... [ 5.928358] systemd[1]: Reached target Swap. [ 5.939096] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 5.967558] systemd[1]: Started Journal Service. [ 6.349701] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 6.932600] random: fast init done [ 7.194603] libata version 3.00 loaded. [ 7.253718] ata_piix 0000:00:01.1: version 2.13 [ 7.265411] scsi host0: ata_piix [ 7.284809] scsi host1: ata_piix [ 7.288162] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.291743] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 7.643678] ip (324) used greatest stack depth: 13080 bytes left [ 7.792881] ip (347) used greatest stack depth: 12336 bytes left [ 14.503435] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 15.302750] systemd-journald[100]: Received SIGTERM from PID 1 (systemd). [ 15.842404] SELinux: Disabled at runtime. [ 15.844560] SELinux: Unregistering netfilter hooks [ 16.016318] ip_tables: (C) 2000-2006 Netfilter Core Team [ 16.036499] systemd[1]: Inserted module 'ip_tables' [ 17.262874] mount.nfs (581) used greatest stack depth: 11856 bytes left [ 17.427101] systemd-journald[570]: Received request to flush runtime journal from PID 1 [ 17.935503] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 17.989412] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.089553] cryptd: max_cpu_qlen set to 1000 [ 18.166408] AVX version of gcm_enc/dec engaged. [ 18.168117] AES CTR mode by8 optimization enabled [ 18.296782] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 18.378434] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 18.392286] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 18.392380] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 18.800065] EDAC MC: Ver: 3.0.0 [ 18.825520] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 18.825537] EDAC sbridge: Ver: 1.1.2 [ 20.473229] hrtimer: interrupt took 2648334 ns [ 23.793766] loadkeys (774) used greatest stack depth: 11824 bytes left [ 23.935441] mount.nfs (779) used greatest stack depth: 10704 bytes left [ 47.603364] device-mapper: uevent: version 1.0.3 [ 47.609642] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 60.160523] libcfs: loading out-of-tree module taints kernel. [ 60.163765] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 60.272736] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 69.053893] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 69.549274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.560637] alg: No test for adler32 (adler32-zlib) [ 70.671709] Lustre: Lustre: Build Version: 2.16.54_76_g2a51fea [ 71.047860] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 71.055498] LNet: Accept secure, port 988 [ 72.693112] Key type lgssc registered [ 73.435089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.672972] icp: module license 'CDDL' taints kernel. [ 80.675130] Disabling lock debugging due to kernel taint [ 83.402052] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 89.307471] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.117462] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 104.214346] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 104.231949] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 109.300038] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 109.313309] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.199742] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 123.975318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 124.014626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 124.037405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 125.238417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 125.265319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 125.326015] Lustre: lustre-MDT0000: new disk, initializing [ 125.401757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 125.417266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 125.491719] mount.lustre (7247) used greatest stack depth: 10112 bytes left [ 127.587565] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 136.666968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.734123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 136.804856] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 136.820175] Lustre: 8411:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 136.840851] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 136.844173] Lustre: Skipped 1 previous similar message [ 136.941408] Lustre: lustre-MDT0001: new disk, initializing [ 136.999559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 137.029720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 137.037979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 138.944993] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 142.118370] random: crng init done [ 142.266779] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 148.064136] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 148.073940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 148.107040] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 148.124368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 148.276781] Lustre: 9616:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 148.291414] Lustre: lustre-OST0000: new disk, initializing [ 148.298065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 148.358625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 150.005467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 150.016597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 150.106071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 151.612580] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.274027] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 161.292093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 161.344177] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 161.351376] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 161.410039] Lustre: 10956:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 161.435814] Lustre: lustre-OST0001: new disk, initializing [ 161.441212] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 161.492382] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 165.075344] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.887334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 168.903768] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 169.027878] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 174.041355] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 177.671285] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 185.803551] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing check_logdir /tmp/testlogs/ [ 187.830797] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing yml_node [ 190.139841] Lustre: DEBUG MARKER: Client: 2.16.54.76 [ 191.847429] Lustre: DEBUG MARKER: MDS: 2.16.54.76 [ 193.672009] Lustre: DEBUG MARKER: OSS: 2.16.54.76 [ 195.015494] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Thu May 1 18:57:46 EDT 2025 [ 208.403695] Lustre: DEBUG MARKER: - need mds1 <= 2.14.55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 209.759858] Lustre: DEBUG MARKER: - need mds1 < v2_14_55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 211.196044] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42c 42b 118c 118d 407 119i 851 411b [ 212.515131] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 [ 213.876483] Lustre: DEBUG MARKER: === sanity: start setup 18:58:05 (1746140285) === [ 216.641993] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing check_config_client /mnt/lustre [ 231.359447] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 233.865858] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 235.859480] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.225757] Lustre: DEBUG MARKER: === sanity: finish setup 18:58:31 (1746140311) === [ 244.188479] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 18:58:35 (1746140315) [ 246.539498] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 247.870782] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 18:58:39 (1746140319) [ 253.820855] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 18:58:45 (1746140325) [ 381.688180] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 19:00:53 (1746140453) [ 386.458806] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 19:00:57 (1746140457) [ 387.415405] Lustre: *** cfs_fail_loc=15b, val=0*** [ 387.423486] Lustre: *** cfs_fail_loc=15b, val=0*** [ 391.788205] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 19:01:03 (1746140463) [ 397.099289] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 19:01:08 (1746140468) [ 397.933981] Lustre: *** cfs_fail_loc=19a, val=0*** [ 398.520944] Lustre: *** cfs_fail_loc=19a, val=0*** [ 399.944180] Lustre: *** cfs_fail_loc=19a, val=0*** [ 399.946726] Lustre: Skipped 1 previous similar message [ 402.501519] Lustre: *** cfs_fail_loc=19a, val=0*** [ 402.508434] Lustre: Skipped 3 previous similar messages [ 406.627598] Lustre: *** cfs_fail_loc=19a, val=0*** [ 406.635593] Lustre: Skipped 5 previous similar messages [ 414.833153] Lustre: *** cfs_fail_loc=19a, val=0*** [ 414.837177] Lustre: Skipped 12 previous similar messages [ 431.334313] Lustre: *** cfs_fail_loc=19a, val=0*** [ 431.345517] Lustre: Skipped 26 previous similar messages [ 463.482354] Lustre: *** cfs_fail_loc=19a, val=0*** [ 463.488584] Lustre: Skipped 46 previous similar messages [ 474.090733] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 19:02:25 (1746140545) [ 474.963575] Lustre: *** cfs_fail_loc=188, val=0*** [ 477.488419] Lustre: *** cfs_fail_loc=189, val=0*** [ 483.961069] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 485.115564] Lustre: DEBUG MARKER: == sanity test 60j: llog_reader reports corruptions ====== 19:02:36 (1746140556) [ 488.835568] Lustre: lustre-MDD0000: changelog on [ 490.685272] Lustre: lustre-MDD0001: changelog on [ 496.784828] Lustre: DEBUG MARKER: SKIP: sanity test_60j path oi.1/0x1:0xb:0x0 is not in 'O/1/d/' format [ 498.651201] Lustre: lustre-MDD0001: changelog off [ 501.106138] Lustre: lustre-MDD0000: changelog off [ 503.169330] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 19:02:54 (1746140574) [ 507.864176] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 19:02:59 (1746140579) [ 512.668437] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 19:03:04 (1746140584) [ 578.712298] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 19:04:10 (1746140650) [ 588.547846] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 19:04:20 (1746140660) [ 594.297106] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 595.476875] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 19:04:27 (1746140667) [ 601.119599] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 19:04:32 (1746140672) [ 636.963345] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 19:05:08 (1746140708) [ 639.553539] Lustre: *** cfs_fail_loc=725, val=0*** [ 644.741053] Lustre: *** cfs_fail_loc=725, val=0*** [ 649.816571] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 19:05:21 (1746140721) [ 656.690866] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 19:05:28 (1746140728) [ 665.223086] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 19:05:37 (1746140737) [ 680.623483] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 19:05:52 (1746140752) [ 686.320668] Lustre: *** cfs_fail_loc=513, val=17*** [ 686.323513] LustreError: 9623:0:(service.c:2266:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1830960644409216 [ 687.623507] Lustre: Failing over lustre-OST0000 [ 687.668363] LustreError: 32212:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 687.809425] Lustre: server umount lustre-OST0000 complete [ 689.780015] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.800518] Lustre: Skipped 1 previous similar message [ 694.791829] LustreError: 17335:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 694.801172] LustreError: 17335:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 697.396837] LustreError: 17335:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 699.797150] LustreError: 9623:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 699.811681] LustreError: 9623:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 702.411212] LustreError: 9623:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 703.966894] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 703.977843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 704.044574] Lustre: 32752:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 704.132635] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 704.144575] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 705.311320] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 707.366479] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 707.369157] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [ 707.775763] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [ 712.767700] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 713.984634] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 721.447185] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 19:06:33 (1746140793) [ 726.006428] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 19:06:37 (1746140797) [ 731.480329] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 19:06:43 (1746140803) [ 735.976519] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 19:06:47 (1746140807) [ 741.377862] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 19:06:52 (1746140812) [ 746.555451] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 19:06:58 (1746140818) [ 751.302970] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 19:07:03 (1746140823) [ 756.893578] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 19:07:08 (1746140828) [ 761.946144] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 19:07:13 (1746140833) [ 768.380027] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 19:07:19 (1746140839) [ 774.422890] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 19:07:26 (1746140846) [ 775.970037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.982109] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 775.988142] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [ 776.000925] Lustre: Skipped 1 previous similar message [ 776.659057] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 777.318027] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 777.336277] Lustre: Skipped 1 previous similar message [ 777.349037] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [ 777.365944] Lustre: Skipped 1 previous similar message [ 777.990665] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 778.000033] Lustre: Skipped 1 previous similar message [ 791.896699] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 804.328862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 804.335687] Lustre: Skipped 1 previous similar message [ 804.340894] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 804.349537] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 804.366004] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [ 804.370487] Lustre: Skipped 1 previous similar message [ 806.897850] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 807.137089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 809.936362] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 810.084862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 823.566561] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 836.038982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.051135] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 836.067510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 836.085136] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [ 837.873234] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 838.044374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 840.060098] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 840.184485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 852.495537] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 864.965757] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.978491] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 864.982655] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 864.988795] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [ 867.350560] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 867.515847] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 869.947165] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 870.105584] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 882.883271] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 894.281664] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 894.291859] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 894.302183] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 894.313722] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [ 896.319868] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 896.485595] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 898.593292] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 898.786309] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 903.164782] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 19:09:35 (1746140975) [ 907.440004] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 19:09:39 (1746140979) [ 912.139459] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 19:09:43 (1746140983) [ 932.648010] Lustre: DEBUG MARKER: == sanity test 65o: pool inheritance for mdt component === 19:10:04 (1746141004) [ 953.230625] Lustre: DEBUG MARKER: == sanity test 65p: setstripe with yaml file and huge number ========================================================== 19:10:25 (1746141025) [ 957.253811] Lustre: DEBUG MARKER: == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 19:10:29 (1746141029) [ 960.825874] Lustre: DEBUG MARKER: == sanity test 65r: prevent all-zero offsets ============= 19:10:32 (1746141032) [ 965.171273] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 19:10:36 (1746141036) [ 973.065751] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 19:10:44 (1746141044) [ 974.087091] Lustre: *** cfs_fail_loc=217, val=0*** [ 975.667351] Lustre: *** cfs_fail_loc=217, val=0*** [ 975.670617] Lustre: Skipped 1 previous similar message [ 980.021767] Lustre: DEBUG MARKER: == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 19:10:51 (1746141051) [ 988.694052] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 990.015960] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 19:11:01 (1746141061) [ 995.674516] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 19:11:07 (1746141067) [ 1001.271084] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 19:11:13 (1746141073) [ 1032.648833] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 19:11:44 (1746141104) [ 1037.103099] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 19:11:48 (1746141108) [ 1041.246892] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 19:11:53 (1746141113) [ 1045.558180] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 19:11:57 (1746141117) [ 1101.718892] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 19:12:53 (1746141173) [ 1140.941290] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 19:13:32 (1746141212) [ 1144.739830] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 19:13:36 (1746141216) [ 1145.120552] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.26@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3829 extent [0-4194303]: client csum 13275fb2, server csum 13275fb1 [ 1146.220686] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1147.474455] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.26@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3829 extent [0-4194303], client returned csum 68931d72 (type 1), server csum ba2880fb (type 1) [ 1148.965077] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1150.404764] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.26@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3829 extent [0-4194303], client returned csum c2589630 (type 2), server csum a9df966e (type 2) [ 1151.943656] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1153.224901] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.26@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3829 extent [0-4194303], client returned csum feff5571 (type 4), server csum 13275fb1 (type 4) [ 1154.623218] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1155.880321] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.26@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3829 extent [0-4194303], client returned csum 6216b068 (type 10), server csum a412b129 (type 10) [ 1157.179447] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1159.675341] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1160.943963] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.26@tcp inode [0x200000407:0xc1f:0x0] object 0x2c0000401:3829 extent [0-4194303], client returned csum 6ff406b8 (type 40), server csum 2c8a05a3 (type 40) [ 1160.958821] LustreError: Skipped 1 previous similar message [ 1162.321434] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1164.644791] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1168.374107] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 19:14:00 (1746141240) [ 1172.174859] Lustre: 9627:0:(tgt_handler.c:1935:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000407:0xc20:0x0]:[0-1048575]-8171d26a-851eadc6 [ 1172.205201] LustreError: dumping log to /tmp/lustre-log.1746141245.9627 [ 1172.296332] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.201.26@tcp inode [0x200000407:0xc20:0x0] object 0x280000401:3840 extent [0-1048575], client returned csum 8171d26a (type 4), server csum 851eadc6 (type 4) [ 1172.304488] LustreError: Skipped 1 previous similar message [ 1184.366442] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 19:14:16 (1746141256) [ 1184.594482] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.201.26@tcp inode [0x200000407:0xc22:0x0] object 0x280000401:3841 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 1191.463060] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 19:14:23 (1746141263) [ 1192.666284] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1192.885936] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.26@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3830 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 1195.473331] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.26@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3830 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 1195.489344] LustreError: Skipped 3 previous similar messages [ 1202.490658] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.26@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3830 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 1202.502344] LustreError: Skipped 3 previous similar messages [ 1213.489381] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.26@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3830 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 1213.501442] LustreError: Skipped 3 previous similar messages [ 1237.832230] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.26@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3830 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 1237.849612] LustreError: Skipped 4 previous similar messages [ 1249.049529] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1270.490866] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.26@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3830 extent [4194304-8388607]: client csum 7d33b9a0, server csum 7d33b99f [ 1270.501311] LustreError: Skipped 15 previous similar messages [ 1305.458264] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1341.965688] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.26@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3830 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 1341.985839] LustreError: Skipped 25 previous similar messages [ 1361.900796] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1419.398324] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1474.811043] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.201.26@tcp inode [0x200000407:0xc23:0x0] object 0x2c0000401:3830 extent [4194304-8388607]: client csum fec05011, server csum fec05010 [ 1474.825963] LustreError: Skipped 47 previous similar messages [ 1475.551068] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1532.120440] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1589.376827] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1593.514864] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 19:21:05 (1746141665) [ 1594.620696] Lustre: *** cfs_fail_loc=21a, val=0*** [ 1596.897514] Lustre: *** cfs_fail_loc=21b, val=0*** [ 1603.139361] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 19:21:14 (1746141674) [ 1603.742573] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1605.175425] Lustre: Modifying parameter lustre.osc.lustre*.checksums in log params [ 1609.111582] Lustre: Disabling parameter lustre.osc.lustre*.checksums in log params [ 1615.085352] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1618.521759] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 19:21:30 (1746141690) [ 1619.581887] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 1620.626128] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1622.879270] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid 50 [ 1623.817521] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid in IDLE state after 0 sec [ 1625.912594] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid 50 [ 1626.907820] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid in FULL state after 0 sec [ 1627.960710] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1630.631808] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid 50 [ 1643.030398] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid in IDLE state after 11 sec [ 1645.137437] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid 50 [ 1646.118044] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid in FULL state after 0 sec [ 1646.998658] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1648.992182] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid 50 [ 1658.700955] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid in IDLE state after 8 sec [ 1661.206346] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid 50 [ 1662.271798] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid in FULL state after 0 sec [ 1663.652114] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1666.038268] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid 50 [ 1678.551759] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid in IDLE state after 11 sec [ 1681.000036] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid 50 [ 1682.048799] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid in FULL state after 0 sec [ 1683.083743] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1685.479267] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid 50 [ 1698.221291] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid in IDLE state after 11 sec [ 1700.650451] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid 50 [ 1701.689231] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid in FULL state after 0 sec [ 1702.748295] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1704.846801] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid 50 [ 1718.518745] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid in IDLE state after 12 sec [ 1721.098738] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid 50 [ 1722.111029] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid in FULL state after 0 sec [ 1723.205892] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1725.790164] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid 50 [ 1738.307373] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid in IDLE state after 11 sec [ 1740.618738] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid 50 [ 1741.583208] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88012a658000.ost_server_uuid in FULL state after 0 sec [ 1746.075987] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1747.078857] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 19:23:39 (1746141819) [ 1751.109106] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 19:23:42 (1746141822) [ 1752.284867] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1753.429336] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1754.492769] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1755.670703] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1759.706651] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1760.746419] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 19:23:52 (1746141832) [ 1766.927714] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 19:23:59 (1746141839) [ 1770.868578] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check =========== 19:24:02 (1746141842) [ 1782.751285] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 19:24:14 (1746141854) [ 1788.242129] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 19:24:20 (1746141860) [ 1789.017592] Lustre: *** cfs_fail_loc=228, val=0*** [ 1792.779898] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 19:24:24 (1746141864) [ 1793.489168] Lustre: *** cfs_fail_loc=228, val=0*** [ 1797.450734] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 19:24:29 (1746141869) [ 1815.354508] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ========================================================== 19:24:47 (1746141887) [ 1820.087370] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 19:24:51 (1746141891) [ 1939.706404] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 19:26:51 (1746142011) [ 1954.895490] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 19:27:06 (1746142026) [ 2000.359835] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 19:27:52 (1746142072) [ 2242.061698] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 19:31:53 (1746142313) [ 2288.680392] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 19:32:40 (1746142360) [ 2294.471232] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 19:32:46 (1746142366) [ 2365.985752] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 19:33:57 (1746142437) [ 2378.166405] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 19:34:09 (1746142449) [ 2384.883503] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 19:34:16 (1746142456) [ 2425.898346] Lustre: DEBUG MARKER: == sanity test 101m: read ahead for small file and last stripe of the file ========================================================== 19:34:57 (1746142497) [ 2434.752008] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 19:35:06 (1746142506) [ 2439.618670] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 19:35:11 (1746142511) [ 2447.202439] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 19:35:19 (1746142519) [ 2451.832361] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 19:35:23 (1746142523) [ 2477.508010] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 19:35:49 (1746142549) [ 2503.020419] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 19:36:14 (1746142574) [ 2504.103467] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 2505.177814] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 2506.353487] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 2507.454613] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2511.717710] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 19:36:23 (1746142583) [ 2513.464573] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2514.671580] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 2515.713351] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 2516.976909] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 2518.087476] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 2522.245380] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 19:36:34 (1746142594) [ 2526.645646] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 19:36:38 (1746142598) [ 2551.876554] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 19:37:03 (1746142623) [ 2557.037260] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 19:37:08 (1746142628) [ 2561.863526] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 19:37:13 (1746142633) [ 2566.215313] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 19:37:17 (1746142637) [ 2570.708367] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 19:37:22 (1746142642) [ 2574.807073] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 19:37:26 (1746142646) [ 2579.124006] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 19:37:30 (1746142650) [ 2583.554844] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 19:37:35 (1746142655) [ 2587.758879] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 19:37:39 (1746142659) [ 2592.879610] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 19:37:44 (1746142664) [ 2763.553720] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 19:40:35 (1746142835) [ 2893.573553] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 19:42:45 (1746142965) [ 2897.954460] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 19:42:49 (1746142969) [ 3029.745693] Lustre: lustre-MDT0000: Client 70d81b31-20bd-4fb7-bd8c-4280c1963518 (at 192.168.201.26@tcp) reconnecting [ 3090.997497] Lustre: lustre-MDT0000: Client 70d81b31-20bd-4fb7-bd8c-4280c1963518 (at 192.168.201.26@tcp) reconnecting [ 3228.110517] Lustre: lustre-MDT0000: Client 70d81b31-20bd-4fb7-bd8c-4280c1963518 (at 192.168.201.26@tcp) reconnecting [ 3658.484568] Lustre: lustre-MDT0000: Client 70d81b31-20bd-4fb7-bd8c-4280c1963518 (at 192.168.201.26@tcp) reconnecting [ 3681.814989] Lustre: 7273:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 30372 > trans_max 3200 [ 3681.830655] Lustre: 7273:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 3681.846584] Lustre: 7273:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28148/0 [ 3681.854734] Lustre: 7273:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 3681.868111] Lustre: 7273:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 3681.885528] Lustre: 7273:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 3681.899275] CPU: 3 PID: 7273 Comm: mdt00_002 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 3681.905980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 3681.914583] Call Trace: [ 3681.916416] [] dump_stack+0x19/0x1b [ 3681.921189] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 3681.928113] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 3681.932795] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 3681.937502] [] lod_trans_start+0x7f/0x2e0 [lod] [ 3681.946024] [] mdd_trans_start+0x14/0x20 [mdd] [ 3681.952297] [] mdd_unlink+0x617/0xea0 [mdd] [ 3681.959430] [] ? finish_task_switch+0x5d/0x1b0 [ 3681.968957] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 3681.974454] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 3681.976696] [] mdt_reint_rec+0x87/0x240 [mdt] [ 3681.984765] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 3681.998052] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 3682.003429] [] mdt_reint+0x67/0x150 [mdt] [ 3682.006945] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 3682.010346] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 3682.015803] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 3682.020670] [] ? put_prev_entity+0x31/0x400 [ 3682.025358] [] ? do_raw_spin_unlock+0x49/0x90 [ 3682.030876] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 3682.040488] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 3682.044878] [] kthread+0xe4/0xf0 [ 3682.046765] [] ? kthread_create_on_node+0x140/0x140 [ 3682.053151] [] ret_from_fork_nospec_begin+0x7/0x21 [ 3682.057967] [] ? kthread_create_on_node+0x140/0x140 [ 3687.385162] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 19:55:59 (1746143759) [ 3689.758482] Lustre: lustre-MDD0000: changelog on [ 3691.340502] Lustre: lustre-MDD0001: changelog on [ 3696.648733] Lustre: lustre-MDD0001: changelog off [ 3698.380305] Lustre: lustre-MDD0000: changelog off [ 3700.161777] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 19:56:12 (1746143772) [ 3700.449862] Lustre: lustre-OST0000: Client 70d81b31-20bd-4fb7-bd8c-4280c1963518 (at 192.168.201.26@tcp) reconnecting [ 3702.590453] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800aa0fb800.ost_server_uuid 50 [ 3703.421893] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa0fb800.ost_server_uuid in FULL state after 0 sec [ 3707.274278] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 19:56:19 (1746143779) [ 3710.527990] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 19:56:22 (1746143782) [ 3711.296395] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [ 3712.314082] Lustre: DEBUG MARKER: == sanity test 104d: runas -u 500 -g 500 lctl dl test ==== 19:56:24 (1746143784) [ 3715.896606] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 19:56:27 (1746143787) [ 3719.605538] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 19:56:31 (1746143791) [ 3723.308423] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 19:56:35 (1746143795) [ 3727.206141] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 19:56:39 (1746143799) [ 3741.206741] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 19:56:53 (1746143813) [ 3745.552809] Lustre: DEBUG MARKER: == sanity test 105f: Enqueue same range flocks =========== 19:56:57 (1746143817) [ 3750.881584] Lustre: DEBUG MARKER: == sanity test 105g: ldlm_lock_debug stack test ========== 19:57:02 (1746143822) [ 3757.546888] Lustre: DEBUG MARKER: == sanity test 105h: Flock functional verify ============= 19:57:09 (1746143829) [ 3762.077304] Lustre: DEBUG MARKER: == sanity test 105i: Flock deadlock verify =============== 19:57:14 (1746143834) [ 3771.485305] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 19:57:23 (1746143843) [ 3775.768000] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 19:57:27 (1746143847) [ 3781.546133] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 19:57:33 (1746143853) [ 3786.125956] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 19:57:37 (1746143857) [ 3967.970900] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 20:00:39 (1746144039) [ 3970.035421] Lustre: *** cfs_fail_loc=147, val=0*** [ 3976.156199] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 20:00:47 (1746144047) [ 3980.713680] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 20:00:52 (1746144052) [ 3984.736690] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 20:00:56 (1746144056) [ 3985.900300] Lustre: *** cfs_fail_loc=217, val=0*** [ 3990.937753] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 3992.069468] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 3993.179653] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 20:01:05 (1746144065) [ 3997.847559] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 20:01:09 (1746144069) [ 4002.560461] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 20:01:14 (1746144074) [ 4003.824901] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4005.181691] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4007.195515] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4010.219588] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4015.196421] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4020.236846] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 20:01:32 (1746144092) [ 4024.557141] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4024.558708] Lustre: Skipped 2 previous similar messages [ 4031.738549] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 20:01:43 (1746144103) [ 4036.594287] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 20:01:48 (1746144108) [ 4040.578494] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4040.582508] Lustre: Skipped 20 previous similar messages [ 4050.341463] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 20:02:02 (1746144122) [ 4054.693203] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 20:02:06 (1746144126) [ 4059.015757] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 20:02:10 (1746144130) [ 4062.227565] LustreError: 9626:0:(ofd_dev.c:1805:ofd_statfs_hdl()) cfs_fail_timeout id 242 sleeping for 10000ms [ 4062.836485] LustreError: 9626:0:(ofd_dev.c:1805:ofd_statfs_hdl()) cfs_fail_timeout interrupted [ 4067.362223] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 20:02:19 (1746144139) [ 4071.884192] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 20:02:23 (1746144143) [ 4076.683584] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 20:02:28 (1746144148) [ 4082.226698] Lustre: DEBUG MARKER: == sanity test 119e: Basic tests of dio read and write at various sizes ========================================================== 20:02:33 (1746144153) [ 4085.660636] Lustre: DEBUG MARKER: sanity test_119e: @@@@@@ FAIL: trivial unaligned dio failed [ 4091.728537] Lustre: DEBUG MARKER: == sanity test 119f: dio vs dio race ===================== 20:02:43 (1746144163) [ 4105.650416] Lustre: DEBUG MARKER: sanity test_119f: @@@@@@ FAIL: dio copy 1 w/bsize 1049600 failed: 1 [ 4112.141340] Lustre: DEBUG MARKER: == sanity test 119g: dio vs buffered I/O race ============ 20:03:03 (1746144183) [ 4130.402357] Lustre: DEBUG MARKER: sanity test_119g: @@@@@@ FAIL: dio copy 1 w/bsize 1049600 failed: 1 [ 4135.994262] Lustre: DEBUG MARKER: == sanity test 119h: basic tests of memory unaligned dio ========================================================== 20:03:27 (1746144207) [ 4137.761956] Lustre: DEBUG MARKER: sanity test_119h: @@@@@@ FAIL: multiop memory unaligned write failed, 1044480 [ 4142.497471] Lustre: DEBUG MARKER: SKIP: sanity test_119i skipping ALWAYS excluded test 119i [ 4143.549680] Lustre: DEBUG MARKER: == sanity test 119j: basic tests of hybrid IO switching == 20:03:35 (1746144215) [ 4144.493748] Lustre: DEBUG MARKER: SKIP: sanity test_119j needs kernel > 4.5.0 for ki_flags support [ 4145.765117] Lustre: DEBUG MARKER: == sanity test 119m: Test DIO readv/writev: exercise iter duplication ========================================================== 20:03:37 (1746144217) [ 4150.245568] Lustre: DEBUG MARKER: == sanity test 119n: Test Unaligned DIO readv() and writev() with unpatched ZFS ========================================================== 20:03:42 (1746144222) [ 4151.298688] Lustre: DEBUG MARKER: SKIP: sanity test_119n need ZFS server without unaligned_dio support [ 4152.302864] Lustre: DEBUG MARKER: == sanity test 119o: Test Unaligned DIO readv() and writev() with unpatched servers ========================================================== 20:03:44 (1746144224) [ 4153.255537] Lustre: DEBUG MARKER: SKIP: sanity test_119o need ldiskfs without unaligned_dio support. [ 4154.290830] Lustre: DEBUG MARKER: == sanity test 119p: Test Unaligned DIO readv() and writev() with patched servers ========================================================== 20:03:46 (1746144226) [ 4155.363643] Lustre: DEBUG MARKER: sanity test_119p: @@@@@@ FAIL: DIO unaligned writev test failed [ 4159.078162] Lustre: DEBUG MARKER: == sanity test 119q: Test patchded Unaligned DIO readv() and writev() ========================================================== 20:03:51 (1746144231) [ 4160.148803] Lustre: DEBUG MARKER: sanity test_119q: @@@@@@ FAIL: DIO unaligned writev test failed: 69631 [ 4163.304437] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 20:03:55 (1746144235) [ 4168.592173] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 20:04:00 (1746144240) [ 4174.444293] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 20:04:06 (1746144246) [ 4180.913987] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 20:04:12 (1746144252) [ 4187.126742] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 20:04:18 (1746144258) [ 4200.360336] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 20:04:32 (1746144272) [ 4214.015699] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 20:04:45 (1746144285) [ 4529.582778] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 20:10:01 (1746144601) [ 4533.223740] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 20:10:05 (1746144605) [ 4539.625897] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 2 sec [ 4541.159440] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 4566.589776] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 15 sec [ 4570.713627] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 3 sec [ 4638.042706] Lustre: DEBUG MARKER: ls -l 3000 files without statahead: 47 sec [ 4648.566629] Lustre: DEBUG MARKER: ls -l 3000 files with statahead: 8 sec [ 4649.492009] Lustre: DEBUG MARKER: ls -l done [ 4690.281417] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity: 39 seconds [ 4696.240657] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 20:12:47 (1746144767) [ 4697.390240] Lustre: DEBUG MARKER: SKIP: sanity test_123ab Test must be statx() syscall supported [ 4698.729838] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 20:12:50 (1746144770) [ 4700.028569] Lustre: DEBUG MARKER: SKIP: sanity test_123ac Test must be statx() syscall supported [ 4701.696093] Lustre: DEBUG MARKER: == sanity test 123ad: Verify batching statahead works correctly ========================================================== 20:12:53 (1746144773) [ 4709.409677] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 2 sec [ 4711.532629] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 4738.069494] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 15 sec [ 4744.290280] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 5 sec [ 4812.993731] Lustre: DEBUG MARKER: ls -l 3000 files without statahead: 47 sec [ 4829.559362] Lustre: DEBUG MARKER: ls -l 3000 files with statahead: 14 sec [ 4830.449629] Lustre: DEBUG MARKER: ls -l done [ 4867.728366] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 36 seconds [ 5270.992480] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 5272.914587] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 5300.449257] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 17 sec [ 5304.361817] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 3 sec [ 5346.457997] Lustre: DEBUG MARKER: ls -l 2000 files without statahead: 31 sec [ 5353.724329] Lustre: DEBUG MARKER: ls -l 2000 files with statahead: 5 sec [ 5354.701655] Lustre: DEBUG MARKER: ls -l done [ 5381.356762] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 25 seconds [ 5654.033139] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 20:28:45 (1746145725) [ 5669.241658] Lustre: DEBUG MARKER: ls done [ 5686.248452] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 20:29:17 (1746145757) [ 5693.309680] Lustre: DEBUG MARKER: == sanity test 123d: Statahead on striped directories works correctly ========================================================== 20:29:24 (1746145764) [ 5708.159652] Lustre: DEBUG MARKER: == sanity test 123e: statahead with large wide striping == 20:29:39 (1746145779) [ 5791.095426] Lustre: 2968:0:(mdt_handler.c:4672:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 5867.180354] Lustre: DEBUG MARKER: == sanity test 123f: Retry mechanism with large wide striping files ========================================================== 20:32:18 (1746145938) [ 5956.057931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5956.070717] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6047.761495] Lustre: lustre-MDT0000: Client 64e9c832-4f3a-4e05-87c8-1d33e1289d9e (at 192.168.201.26@tcp) reconnecting [ 6111.201034] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6111.855645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6229.663368] Lustre: lustre-MDT0000: Client 64e9c832-4f3a-4e05-87c8-1d33e1289d9e (at 192.168.201.26@tcp) reconnecting [ 6344.633892] Lustre: 9717:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 19842 > trans_max 3200 [ 6344.638417] Lustre: 9717:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 6344.642706] Lustre: 9717:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1301/1301/0, xattr_set: 1951/18320/0 [ 6344.646659] Lustre: 9717:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 6344.652111] Lustre: 9717:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 6344.655919] Lustre: 9717:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 6344.661198] CPU: 1 PID: 9717 Comm: mdt00_007 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 6344.665687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 6344.669538] Call Trace: [ 6344.671522] [] dump_stack+0x19/0x1b [ 6344.675307] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 6344.678444] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 6344.682378] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 6344.685971] [] lod_trans_start+0x7f/0x2e0 [lod] [ 6344.688796] [] mdd_trans_start+0x14/0x20 [mdd] [ 6344.691715] [] mdd_unlink+0x617/0xea0 [mdd] [ 6344.696375] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 6344.700254] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 6344.701927] [] mdt_reint_rec+0x87/0x240 [mdt] [ 6344.704317] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 6344.706525] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 6344.709714] [] mdt_reint+0x67/0x150 [mdt] [ 6344.711179] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 6344.713833] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 6344.717710] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 6344.721788] [] ? do_raw_spin_unlock+0x49/0x90 [ 6344.724667] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 6344.727353] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 6344.730358] [] kthread+0xe4/0xf0 [ 6344.732300] [] ? kthread_create_on_node+0x140/0x140 [ 6344.735138] [] ret_from_fork_nospec_begin+0x7/0x21 [ 6344.740173] [] ? kthread_create_on_node+0x140/0x140 [ 6404.858970] Lustre: 4770:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 30372 > trans_max 3200 [ 6404.863741] Lustre: 4770:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 6404.868718] Lustre: 4770:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28148/0 [ 6404.874644] Lustre: 4770:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 6404.881549] Lustre: 4770:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 6404.888319] Lustre: 4770:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 6404.894709] CPU: 0 PID: 4770 Comm: mdt00_011 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 6404.900696] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 6404.905278] Call Trace: [ 6404.906480] [] dump_stack+0x19/0x1b [ 6404.909774] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 6404.916399] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 6404.919890] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 6404.922951] [] lod_trans_start+0x7f/0x2e0 [lod] [ 6404.924884] [] mdd_trans_start+0x14/0x20 [mdd] [ 6404.927884] [] mdd_unlink+0x617/0xea0 [mdd] [ 6404.930837] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 6404.933908] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 6404.938727] [] mdt_reint_rec+0x87/0x240 [mdt] [ 6404.942289] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 6404.945731] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 6404.948932] [] mdt_reint+0x67/0x150 [mdt] [ 6404.952350] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 6404.957279] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 6404.960242] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 6404.962865] [] ? do_raw_spin_unlock+0x49/0x90 [ 6404.965719] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 6404.969858] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 6404.975186] [] kthread+0xe4/0xf0 [ 6404.979726] [] ? kthread_create_on_node+0x140/0x140 [ 6404.982973] [] ret_from_fork_nospec_begin+0x7/0x21 [ 6404.999002] [] ? kthread_create_on_node+0x140/0x140 [ 6430.114059] Lustre: DEBUG MARKER: == sanity test 123g: Test for stat-ahead advise ========== 20:41:41 (1746146501) [ 6456.817764] Lustre: DEBUG MARKER: == sanity test 123h: Verify statahead work with the fname pattern via du ========================================================== 20:42:08 (1746146528) [ 7595.117537] Lustre: DEBUG MARKER: == sanity test 123i: Verify statahead work with the fname indexing pattern ========================================================== 21:01:06 (1746147666) [ 7699.114353] Lustre: DEBUG MARKER: == sanity test 123j: -ENOENT error from batched statahead be handled correctly ========================================================== 21:02:50 (1746147770) [ 7705.016753] Lustre: DEBUG MARKER: == sanity test 123k: Verify statahead work with mdtest shared stat() mode ========================================================== 21:02:56 (1746147776) [ 7706.017647] Lustre: DEBUG MARKER: SKIP: sanity test_123k mdtest not found [ 7707.184713] Lustre: DEBUG MARKER: == sanity test 123l: Avoid panic when revalidate a local cached entry ========================================================== 21:02:58 (1746147778) [ 7783.750290] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 21:04:15 (1746147855) [ 7784.957593] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 7815.572316] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a738a800 [ 7816.616462] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a738a800 [ 7817.909460] Lustre: DEBUG MARKER: LRU=1015 [ 7819.161717] Lustre: DEBUG MARKER: LIMIT=46624 [ 7820.598321] Lustre: DEBUG MARKER: LVF=5512100 [ 7821.745298] Lustre: DEBUG MARKER: OLD_LVF=100 [ 7822.881899] Lustre: DEBUG MARKER: Sleep 50 sec [ 7874.522753] Lustre: DEBUG MARKER: Dropped 441 locks in 50s [ 7875.992614] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 7901.714995] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 21:06:13 (1746147973) [ 7995.454337] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 8120.998027] Lustre: DEBUG MARKER: ls -la time: 124 seconds [ 8122.370775] Lustre: DEBUG MARKER: lru_size = 400 [ 8299.959090] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 8346.470175] Lustre: DEBUG MARKER: ls -la time: 44 seconds [ 8347.714523] Lustre: DEBUG MARKER: lru_size = 4052 [ 8348.974414] Lustre: DEBUG MARKER: ls -la is 64% faster with lru resize enabled [ 8412.427721] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 21:14:43 (1746148483) [ 8443.094809] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize disabled ========================================================== 21:15:14 (1746148514) [ 8475.081097] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 21:15:46 (1746148546) [ 8480.994612] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 21:15:52 (1746148552) [ 8486.936148] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 21:15:58 (1746148558) [ 8493.392755] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 21:16:04 (1746148564) [ 8499.167321] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [ 8522.747595] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 21:16:34 (1746148594) [ 8528.374603] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 21:16:39 (1746148599) [ 8544.404738] Lustre: 20324:0:(osd_handler.c:600:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513281, FID: [0x20000040c:0x82eb:0x0]) is approaching max size limit [ 8545.021884] Lustre: 20324:0:(osd_handler.c:600:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513281, FID: [0x20000040c:0x82eb:0x0]) is approaching max size limit [ 8555.029928] Lustre: 20324:0:(osd_handler.c:596:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513281, FID: [0x20000040c:0x82eb:0x0]) has reached max size limit [ 8580.961694] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 21:17:32 (1746148652) [ 8587.081465] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 21:17:38 (1746148658) [ 8593.922120] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 21:17:45 (1746148665) [ 8599.852265] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 21:17:51 (1746148671) [ 8600.965951] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [ 8602.425848] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 21:17:54 (1746148674) [ 8653.951595] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 21:18:45 (1746148725) [ 8659.158381] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 21:18:50 (1746148730) [ 8687.083017] Lustre: 4770:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 6372 > trans_max 3200 [ 8687.100446] Lustre: 4770:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 8687.105018] Lustre: 4770:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 403/403/0, xattr_set: 604/5748/0 [ 8687.115983] Lustre: 4770:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 20/109/0, punch: 0/0/0, quota 0/0/0 [ 8687.121756] Lustre: 4770:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 8687.129200] Lustre: 4770:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 8687.136485] CPU: 0 PID: 4770 Comm: mdt00_011 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [ 8687.140841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 8687.144002] Call Trace: [ 8687.145571] [] dump_stack+0x19/0x1b [ 8687.148031] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [ 8687.151117] [] top_trans_start+0x7d9/0xab0 [ptlrpc] [ 8687.154638] [] ? lod_declare_ref_add+0x16/0x20 [lod] [ 8687.157219] [] lod_trans_start+0x7f/0x2e0 [lod] [ 8687.159846] [] mdd_trans_start+0x14/0x20 [mdd] [ 8687.162487] [] mdd_unlink+0x617/0xea0 [mdd] [ 8687.164978] [] ? mdd_unlink_sanity_check+0x120/0x120 [mdd] [ 8687.168563] [] mdt_reint_unlink+0xbb7/0x15b0 [mdt] [ 8687.171411] [] mdt_reint_rec+0x87/0x240 [mdt] [ 8687.174132] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [ 8687.176872] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [ 8687.180138] [] mdt_reint+0x67/0x150 [mdt] [ 8687.182727] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [ 8687.186224] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [ 8687.190261] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [ 8687.192718] [] ? do_raw_spin_unlock+0x49/0x90 [ 8687.195512] [] ? _raw_spin_unlock_irq+0xe/0x30 [ 8687.198374] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [ 8687.201276] [] kthread+0xe4/0xf0 [ 8687.203583] [] ? kthread_create_on_node+0x140/0x140 [ 8687.206533] [] ret_from_fork_nospec_begin+0x7/0x21 [ 8687.209606] [] ? kthread_create_on_node+0x140/0x140 [ 8692.978624] Lustre: DEBUG MARKER: == sanity test 130h: FIEMAP deadlock ===================== 21:19:23 (1746148763) [ 8706.491130] Lustre: DEBUG MARKER: == sanity test 130i: FIEMAP (DoM file) =================== 21:19:37 (1746148777) [ 8752.050756] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 21:20:23 (1746148823) [ 8758.295590] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 21:20:29 (1746148829) [ 8764.974273] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 21:20:36 (1746148836) [ 8770.457196] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 21:20:42 (1746148842) [ 8775.771484] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 21:20:47 (1746148847) [ 8780.944770] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 21:20:52 (1746148852) [ 8796.045837] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 21:21:07 (1746148867) [ 8808.627516] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 21:21:19 (1746148879) [ 8830.560888] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 21:21:41 (1746148901) [ 8858.974894] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 21:22:10 (1746148930) [ 8869.091878] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 21:22:20 (1746148940) [ 8885.374003] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8885.378086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8885.389268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8887.703338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8887.707967] Lustre: Skipped 3 previous similar messages [ 8889.184528] LustreError: 25821:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8889.189317] LustreError: 25821:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8889.361601] Lustre: server umount lustre-MDT0000 complete [ 8892.727581] LustreError: 7271:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8892.740880] LustreError: 7271:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 8895.267850] LustreError: 5760:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746148968 with bad export cookie 1714858873530127891 [ 8895.268510] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8895.299951] LustreError: 5760:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8895.425403] LustreError: 26424:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8895.428796] LustreError: 26424:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8895.629716] Lustre: server umount lustre-MDT0001 complete [ 8911.270223] LustreError: 27020:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8911.287572] LustreError: 27020:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8911.468342] Lustre: server umount lustre-OST0000 complete [ 8913.764249] Lustre: 3790:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746148970/real 1746148970] req@ffff880099459c00 x1830960712814336/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746148986 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 8913.789273] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8913.803964] Lustre: Skipped 3 previous similar messages [ 8916.232990] Lustre: 3792:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746148973/real 1746148973] req@ffff88006f086300 x1830960712814720/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746148989 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 8917.855054] LustreError: 27630:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8917.867292] LustreError: 27630:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8918.223874] Lustre: server umount lustre-OST0001 complete [ 8923.200526] device-mapper: core: cleaned up [ 8931.149731] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 8933.365434] Key type lgssc unregistered [ 8933.581245] LNet: 28491:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8934.599452] LNet: Removed LNI 192.168.201.126@tcp [ 8945.254944] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 8946.123673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8946.156898] alg: No test for adler32 (adler32-zlib) [ 8947.128993] Lustre: Lustre: Build Version: 2.16.54_76_g2a51fea [ 8947.305512] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 8947.313727] LNet: Accept secure, port 988 [ 8948.932379] Key type lgssc registered [ 8949.841728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8961.335298] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 8964.894733] device-mapper: uevent: version 1.0.3 [ 8964.900058] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 8970.556808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8970.590737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8972.147163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8972.245655] mount.lustre (30693) used greatest stack depth: 9600 bytes left [ 8975.131166] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [ 8982.148586] LustreError: 30718:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8982.156855] LustreError: 30718:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 8984.235277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8984.316249] Lustre: 31790:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8984.681754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8984.809018] mount.lustre (31790) used greatest stack depth: 9584 bytes left [ 8987.196640] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [ 8989.507394] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 8996.404535] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 8996.415721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8996.582345] Lustre: 460:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8996.773469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8996.824747] mount.lustre (460) used greatest stack depth: 9328 bytes left [ 8997.765930] LustreError: 466:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8997.778699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:21374 to 0x280000403:21505) [ 9000.248099] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [ 9002.777077] LustreError: 603:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9007.804449] LustreError: 464:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9008.310764] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 9008.321154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9008.390006] Lustre: 1689:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9008.538301] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9009.542397] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:10422 to 0x2c0000400:10465) [ 9009.542715] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:10575 to 0x280000400:10593) [ 9009.548468] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:22228 to 0x2c0000403:22401) [ 9011.940747] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [ 9019.333195] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9021.797723] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 9033.283792] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 21:25:04 (1746149104) [ 9073.752537] LNet: 4428:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 9073.815976] Lustre: DEBUG MARKER:  [ 9073.817510] Lustre: DEBUG MARKER:  [ 9074.485975] LNet: 4530:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 9074.491014] LNet: 4530:0:(debug.c:388:cfs_str2mask()) Skipped 1 previous similar message [ 9113.092889] LNet: 6063:0:(debug.c:388:cfs_str2mask()) unknown mask ''. mask usage: [+|-] ... [ 9113.099121] LNet: 6063:0:(debug.c:388:cfs_str2mask()) Skipped 1 previous similar message [ 9113.131463] Lustre: DEBUG MARKER:  [ 9113.133299] Lustre: DEBUG MARKER:  [ 9119.750056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9119.753059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9119.769899] Lustre: Skipped 2 previous similar messages [ 9123.623165] LustreError: 6341:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9123.686967] Lustre: server umount lustre-MDT0000 complete [ 9124.762065] LustreError: 30718:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9124.767704] LustreError: 30718:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 9129.494913] LustreError: 31809:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746149202 with bad export cookie 12550799378640570117 [ 9129.501430] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9129.517809] LustreError: 31809:0:(ldlm_lockd.c:2549:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9129.649122] LustreError: 6944:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 9129.652988] LustreError: 6944:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9129.830340] Lustre: server umount lustre-MDT0001 complete [ 9145.597707] Lustre: 28912:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746149202/real 1746149202] req@ffff880083c02a00 x1830969954903296/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746149218 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 9145.612510] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9145.620782] Lustre: Skipped 1 previous similar message [ 9145.642986] LustreError: 7542:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 9145.646333] LustreError: 7542:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9145.685088] Lustre: server umount lustre-OST0000 complete [ 9150.338487] LustreError: 8146:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9150.343884] LustreError: 8146:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9150.461930] Lustre: server umount lustre-OST0001 complete [ 9154.143362] device-mapper: core: cleaned up [ 9159.314673] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 9160.769756] Key type lgssc unregistered [ 9160.926184] LNet: 9069:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9161.940632] LNet: Removed LNI 192.168.201.126@tcp [ 9171.416559] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 9171.719084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9171.727386] alg: No test for adler32 (adler32-zlib) [ 9172.691552] Lustre: Lustre: Build Version: 2.16.54_76_g2a51fea [ 9172.852947] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 9172.858406] LNet: Accept secure, port 988 [ 9174.465561] Key type lgssc registered [ 9175.147658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9185.427907] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 9188.955584] device-mapper: uevent: version 1.0.3 [ 9188.966066] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 9193.446312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9193.476609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9194.844040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9196.921043] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [ 9203.578437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9203.661526] Lustre: 12361:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9203.940283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9205.974064] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [ 9207.922272] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 9214.505152] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 9214.515300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9214.692421] Lustre: 13490:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9214.874861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9218.662969] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [ 9219.874267] LustreError: 13495:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9219.888817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:21374 to 0x280000403:21537) [ 9219.930986] LustreError: 13495:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 9224.889938] LustreError: 13495:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9227.437821] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 9227.453479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9227.592124] Lustre: 14693:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9227.861741] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9228.859261] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:10422 to 0x2c0000400:10497) [ 9228.871095] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:10575 to 0x280000400:10625) [ 9228.871353] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:22228 to 0x2c0000403:22433) [ 9231.902047] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [ 9238.800328] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9241.599046] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 9252.281579] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 21:28:43 (1746149323) [10024.267065] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 21:41:35 (1746150095) [10040.170601] Lustre: *** cfs_fail_loc=327, val=500*** [10073.152217] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 21:42:24 (1746150144) [10078.890229] Lustre: *** cfs_fail_loc=328, val=500*** [10078.897557] Lustre: Skipped 1025 previous similar messages [10079.894668] Lustre: *** cfs_fail_loc=328, val=500*** [10079.898265] Lustre: Skipped 93 previous similar messages [10081.798631] Lustre: 14462:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff88009a350e00 x1830970183497088/t0(0) o101->0ec1d171-30ff-4565-b8a1-cb97de5ee004@192.168.201.26@tcp:510/0 lens 648/0 e 0 to 0 dl 1746150165 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [10082.853638] Lustre: *** cfs_fail_loc=328, val=500*** [10082.856295] Lustre: Skipped 158 previous similar messages [10082.859131] Lustre: 14462:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff880077a49500 x1830970183497856/t0(0) o101->0ec1d171-30ff-4565-b8a1-cb97de5ee004@192.168.201.26@tcp:511/0 lens 648/0 e 0 to 0 dl 1746150166 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [10083.910737] Lustre: 14462:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff880077a4aa00 x1830970183498880/t0(0) o101->0ec1d171-30ff-4565-b8a1-cb97de5ee004@192.168.201.26@tcp:512/0 lens 648/0 e 0 to 0 dl 1746150167 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [10086.337384] Lustre: 11284:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff880077e25f80 x1830970183562624/t0(0) o101->0ec1d171-30ff-4565-b8a1-cb97de5ee004@192.168.201.26@tcp:515/0 lens 648/0 e 0 to 0 dl 1746150170 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [10087.384303] Lustre: *** cfs_fail_loc=328, val=500*** [10087.388221] Lustre: Skipped 125 previous similar messages [10091.020423] Lustre: 11284:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff880063ecce00 x1830970183596800/t0(0) o101->0ec1d171-30ff-4565-b8a1-cb97de5ee004@192.168.201.26@tcp:520/0 lens 648/0 e 0 to 0 dl 1746150175 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 [10091.045560] Lustre: 11284:0:(ldlm_lockd.c:1320:ldlm_handle_enqueue()) Skipped 3 previous similar messages [10096.230957] Lustre: *** cfs_fail_loc=328, val=500*** [10096.249198] Lustre: Skipped 67 previous similar messages [10099.223221] LustreError: 23839:0:(ldlm_lockd.c:3258:lock_reclaim_threshold_mb_store()) Failed to set lock_reclaim_threshold_mb, rc = -22. [10115.087779] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [10116.666592] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [10118.180740] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 21:43:09 (1746150189) [10156.481073] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 21:43:48 (1746150228) [10198.883978] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 21:44:30 (1746150270) [10228.759651] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 21:45:00 (1746150300) [10258.634785] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 21:45:30 (1746150330) [10275.605778] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 21:45:46 (1746150346) [10295.232077] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 21:46:06 (1746150366) [10310.474224] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 21:46:22 (1746150382) [10319.707870] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 21:46:31 (1746150391) [10341.228387] Lustre: DEBUG MARKER: == sanity test 150h: Verify extend fallocate updates the file size ========================================================== 21:46:52 (1746150412) [10342.966536] Lustre: DEBUG MARKER: SKIP: sanity test_150h Test must be statx() syscall supported [10344.224858] Lustre: DEBUG MARKER: == sanity test 150ia: Verify fallocate zero-range ZERO functionality ========================================================== 21:46:55 (1746150415) [10358.697701] Lustre: DEBUG MARKER: == sanity test 150ib: Verify fallocate zero-range PREALLOC functionality ========================================================== 21:47:10 (1746150430) [10376.750870] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 21:47:28 (1746150448) [10390.252262] bash (1612): drop_caches: 1 [10400.150251] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 21:47:51 (1746150471) [10406.600593] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 21:47:57 (1746150477) [10412.444109] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 21:48:04 (1746150484) [10418.322486] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 21:48:09 (1746150489) [10423.761670] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 21:48:15 (1746150495) [10424.954620] LustreError: 15668:0:(fld_handler.c:244:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [10431.970590] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 21:48:23 (1746150503) [10433.396097] LustreError: 14462:0:(fld_handler.c:244:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [10433.409796] LustreError: 14462:0:(fld_handler.c:244:fld_server_lookup()) Skipped 3 previous similar messages [10440.383176] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 21:48:31 (1746150511) [10446.130663] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 21:48:37 (1746150517) [10452.587163] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 21:48:44 (1746150524) [10457.978238] Lustre: DEBUG MARKER: == sanity test 154ea: .lustre is not returned by readdir (2) ========================================================== 21:48:49 (1746150529) [10508.358252] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 21:49:39 (1746150579) [10514.492077] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 21:49:46 (1746150586) [11307.354811] Lustre: DEBUG MARKER: == sanity test 154h: Verify interactive path2fid ========= 22:02:58 (1746151378) [11314.255066] Lustre: DEBUG MARKER: == sanity test 154i: fid2path for path longer than PATH_MAX ========================================================== 22:03:05 (1746151385) [11349.163552] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 22:03:40 (1746151420) [11359.178667] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 22:03:50 (1746151430) [11369.619720] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 22:04:01 (1746151441) [11379.916302] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 22:04:11 (1746151451) [11390.436254] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 22:04:22 (1746151462) [11432.538319] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 22:05:04 (1746151504) [11468.218215] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 22:05:39 (1746151539) [11505.952696] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 22:06:17 (1746151577) [11545.763275] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 22:06:57 (1746151617) [11552.545031] Lustre: DEBUG MARKER: Turn on read and write cache [11555.179814] Lustre: DEBUG MARKER: Write data and read it back. [11556.124636] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [11559.157401] Lustre: DEBUG MARKER: cache hits: before: 65582, after: 65585 [11560.222793] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [11562.171890] Lustre: DEBUG MARKER: cache hits:: before: 65585, after: 65588 [11563.129355] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [11566.124293] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [11569.317719] Lustre: DEBUG MARKER: cache hits:: before: 65588, after: 65591 [11570.901906] Lustre: DEBUG MARKER: Write data and read it back. [11572.196332] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [11575.454969] Lustre: DEBUG MARKER: cache hits:: before: 65591, after: 65594 [11576.686258] Lustre: DEBUG MARKER: Turn off read and write cache [11579.531805] Lustre: DEBUG MARKER: Write data and read it back [11580.750392] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [11583.921940] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65594 [11585.271187] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [11588.089853] Lustre: DEBUG MARKER: Write data and read it back [11589.257549] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [11592.335100] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65594 [11593.589227] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [11596.428907] Lustre: DEBUG MARKER: cache hits:: before: 65594, after: 65597 [11602.225549] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 22:07:53 (1746151673) [11604.104072] Lustre: lustre-MDD0000: changelog on [11606.021252] Lustre: lustre-MDD0001: changelog on [11617.089933] Lustre: Failing over lustre-MDT0000 [11617.250809] LustreError: 17789:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11617.345517] Lustre: server umount lustre-MDT0000 complete [11617.819276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11617.827782] LustreError: 12483:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11622.820664] LustreError: 12483:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11622.850938] LustreError: 12483:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 7 previous similar messages [11622.908235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11623.062340] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11623.281296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11623.327586] Lustre: lustre-MDD0000: changelog on [11623.355848] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11624.733288] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11625.400351] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [11628.283547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [11628.669483] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11628.703831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:22376 to 0x280000403:22401) [11628.707406] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23270 to 0x2c0000403:23297) [11629.147145] Lustre: lustre-MDD0000: changelog off [11630.110592] Lustre: lustre-MDD0001: changelog off [11638.356398] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 22:08:30 (1746151710) [11640.063013] Lustre: lustre-MDD0000: changelog on [11647.210099] Lustre: lustre-MDD0001: changelog off [11650.630654] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 22:08:42 (1746151722) [11652.233194] Lustre: lustre-MDD0000: changelog on [11652.235633] Lustre: Skipped 1 previous similar message [11660.958415] Lustre: lustre-MDD0001: changelog off [11660.962932] Lustre: Skipped 1 previous similar message [11664.801226] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 22:08:56 (1746151736) [11667.074535] Lustre: lustre-MDD0000: changelog on [11667.081575] Lustre: Skipped 1 previous similar message [11675.365074] Lustre: lustre-MDD0001: changelog off [11675.367569] Lustre: Skipped 1 previous similar message [11679.040685] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 22:09:10 (1746151750) [11683.522934] Lustre: lustre-MDD0001: changelog on [11683.525544] Lustre: Skipped 2 previous similar messages [11689.930973] Lustre: lustre-MDD0001: changelog off [11689.933493] Lustre: Skipped 1 previous similar message [11693.490472] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 22:09:25 (1746151765) [11702.867392] Lustre: DEBUG MARKER: 1746151774: creating first dirs [11722.913476] Lustre: *** cfs_fail_loc=1313, val=3*** [11722.916149] Lustre: 14462:0:(mdd_dir.c:911:mdd_changelog_emrg_cleanup()) lustre-MDD0000: changelog has only 3 free catalog entries [11722.925864] Lustre: 14462:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [11722.938765] Lustre: 26289:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl7 idle for 23s with 4 unprocessed records [11736.900893] Lustre: lustre-MDD0001: changelog off [11736.902762] Lustre: Skipped 1 previous similar message [11740.549872] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 22:10:12 (1746151812) [11742.653798] Lustre: lustre-MDD0000: changelog on [11742.656184] Lustre: Skipped 2 previous similar messages [11755.888521] Lustre: 14463:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [11755.892555] Lustre: 14463:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [11755.906148] Lustre: 29591:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl9 idle for 10s with 4 unprocessed records [11755.915652] Lustre: 29591:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [11769.752556] Lustre: lustre-MDD0000: changelog off [11769.755716] Lustre: Skipped 2 previous similar messages [11771.390892] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 22:10:43 (1746151843) [11797.023801] Lustre: *** cfs_fail_loc=1316, val=0*** [11797.026610] Lustre: 14463:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0001: simulate starting changelog garbage collection [11797.042227] Lustre: 14463:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [11797.058836] Lustre: 432:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0001: force deregister of changelog user cl11 idle for 20s with 3 unprocessed records [11797.073589] Lustre: 432:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [11798.175418] Lustre: Failing over lustre-MDT0000 [11798.282606] Lustre: Failing over lustre-MDT0001 [11798.584414] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11798.585876] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11798.619204] Lustre: Skipped 5 previous similar messages [11800.500131] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.26@tcp (stopping) [11800.504277] Lustre: Skipped 3 previous similar messages [11802.415187] LustreError: 725:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [11802.419633] LustreError: 725:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11802.471892] Lustre: server umount lustre-MDT0001 complete [11802.674708] Lustre: server umount lustre-MDT0000 complete [11807.558139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11807.663219] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11807.908023] LustreError: 13505:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11807.997786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11808.024837] Lustre: lustre-MDD0000: changelog on [11808.027015] Lustre: Skipped 3 previous similar messages [11808.030849] Lustre: 1998:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 33 to index 34, being cleared now [11808.050859] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11810.076848] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [11813.004033] LustreError: 13505:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11815.647212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11815.726265] Lustre: 2887:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [11815.862264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [11815.997646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11816.037489] Lustre: 2887:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0001 : orphan changelog records found, starting from index 13 to index 14, being cleared now [11816.051717] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [11818.168482] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [11818.994866] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [11818.997356] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [11818.997357] Lustre: Skipped 3 previous similar messages [11819.005362] Lustre: Skipped 1 previous similar message [11820.697262] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11820.699376] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [11820.699378] Lustre: Skipped 3 previous similar messages [11820.748268] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:10502 to 0x2c0000400:10529) [11820.748436] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:10631 to 0x280000400:10657) [11821.057169] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23270 to 0x2c0000403:23329) [11821.058146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:22403 to 0x280000403:22433) [11833.542351] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 22:11:45 (1746151905) [11840.305932] Lustre: lustre-MDD0000: changelog off [11840.310233] Lustre: Skipped 2 previous similar messages [11840.315258] LustreError: 5443:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_race id 1315 sleeping [11842.870577] LustreError: 5612:0:(mdd_device.c:1746:mdd_changelog_user_register()) cfs_fail_race id 1315 waking [11842.877365] LustreError: 5443:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_fail_race id 1315 awake: rc=2446 [11844.108239] LustreError: 5863:0:(mdd_device.c:2021:mdd_changelog_user_purge()) cfs_fail_race id 1315 waking [11859.086094] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 22:12:10 (1746151930) [11874.203843] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 22:12:25 (1746151945) [11879.167969] LustreError: 7249:0:(mdd_dir.c:966:mdd_changelog_store()) cfs_fail_timeout id 15d sleeping for 3000ms [11882.186124] LustreError: 7249:0:(mdd_dir.c:966:mdd_changelog_store()) cfs_fail_timeout id 15d awake [11894.699999] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 22:12:46 (1746151966) [11910.182901] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 22:13:02 (1746151982) [11920.288230] LustreError: 7249:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_race id 15f sleeping [11922.288381] LustreError: 2007:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_fail_race id 15f waking [11922.301839] LustreError: 2007:0:(mdd_device.c:394:llog_changelog_cancel_cb()) Skipped 1 previous similar message [11922.324014] LustreError: 7249:0:(mdd_device.c:394:llog_changelog_cancel_cb()) cfs_fail_race id 15f awake: rc=2975 [11932.534553] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 22:13:24 (1746152004) [13456.552833] LustreError: 2009:0:(mdd_device.c:408:llog_changelog_cancel_cb()) cfs_race id 16c sleeping [13458.560119] LustreError: 7249:0:(llog_osd.c:1136:llog_osd_next_block()) cfs_fail_race id 16c waking [13458.564869] LustreError: 2009:0:(mdd_device.c:408:llog_changelog_cancel_cb()) cfs_fail_race id 16c awake: rc=2995 [13469.257128] Lustre: lustre-MDD0001: changelog off [13469.259089] Lustre: Skipped 11 previous similar messages [13473.408476] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 22:39:04 (1746153544) [13475.647982] Lustre: lustre-MDD0000: changelog on [13475.651267] Lustre: Skipped 15 previous similar messages [13478.394591] LustreError: 17100:0:(mdd_device.c:1700:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [13479.021395] Lustre: 17165:0:(mdd_device.c:1717:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [13479.716209] LustreError: 17229:0:(mdd_device.c:1709:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [13496.662103] Lustre: lustre-MDD0000: changelog off [13496.665584] Lustre: Skipped 1 previous similar message [13501.556461] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 22:39:33 (1746153573) [13503.710820] Lustre: lustre-MDD0000: changelog on [13503.718793] Lustre: Skipped 1 previous similar message [13509.081225] Lustre: Failing over lustre-MDT0000 [13509.218704] LustreError: 19690:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [13509.222696] LustreError: 19690:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [13509.299894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13509.304495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13509.316651] Lustre: Skipped 1 previous similar message [13509.326023] LustreError: 26293:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13509.410415] Lustre: server umount lustre-MDT0000 complete [13511.328605] LustreError: 3417:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13512.138611] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13512.162604] Lustre: Skipped 1 previous similar message [13515.178508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13515.320424] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13515.549676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13515.611980] Lustre: 20353:0:(mdd_device.c:621:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 90156 to index 18446744073709551615, being cleared now [13515.645022] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13516.321519] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [13517.758952] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [13520.542414] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [13520.549734] Lustre: Skipped 1 previous similar message [13520.570835] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [13520.578442] Lustre: Skipped 1 previous similar message [13520.624639] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:23270 to 0x2c0000403:23361) [13520.625581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:22435 to 0x280000403:22465) [13529.177090] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 22:40:00 (1746153600) [13532.329766] Lustre: lustre-MDD0000: changelog off [13532.334629] Lustre: Skipped 2 previous similar messages [13536.863072] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records * time ========================================================== 22:40:08 (1746153608) [13539.866357] Lustre: lustre-MDD0000: changelog on [13539.870736] Lustre: Skipped 2 previous similar messages [13548.485619] Lustre: 7249:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [13548.490310] Lustre: 7249:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [13548.498560] Lustre: 22945:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl2 idle for 864009s with 500000004 unprocessed records [13548.510998] Lustre: 22945:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [13562.389230] Lustre: DEBUG MARKER: == sanity test 160t: changelog garbage collect on lack of space ========================================================== 22:40:34 (1746153634) [13604.057377] Lustre: *** cfs_fail_loc=18c, val=1211180*** [13604.060240] Lustre: Skipped 1 previous similar message [13604.062562] Lustre: 2008:0:(mdd_dir.c:881:mdd_changelog_is_space_safe()) lustre-MDD0000: changelog uses 21MB with 1MB space limit [13604.067824] Lustre: 2008:0:(mdd_dir.c:994:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [13604.072503] Lustre: 2008:0:(mdd_dir.c:994:mdd_changelog_store()) Skipped 1 previous similar message [13604.081141] Lustre: 25606:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl3-user1 idle for 40s with 7506 unprocessed records [13604.089979] Lustre: 25606:0:(mdd_trans.c:150:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [13604.889554] Lustre: *** cfs_fail_loc=18c, val=1211180*** [13616.009087] Lustre: lustre-MDD0000: changelog off [13616.011964] Lustre: Skipped 2 previous similar messages [13635.635760] Lustre: DEBUG MARKER: == sanity test 160u: changelog rename record type name and sname strings are correct ========================================================== 22:41:47 (1746153707) [13638.863215] Lustre: lustre-MDD0000: changelog on [13638.865238] Lustre: Skipped 3 previous similar messages [13649.094463] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 22:42:00 (1746153720) [13672.925880] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 22:42:24 (1746153744) [13698.586262] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 22:42:50 (1746153770) [13714.316303] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 22:43:06 (1746153786) [13729.438992] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 22:43:21 (1746153801) [13735.419878] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 22:43:27 (1746153807) [13740.574864] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 22:43:32 (1746153812) [13824.375567] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 22:44:56 (1746153896) [13831.563211] Lustre: Failing over lustre-OST0000 [13831.641755] LustreError: 429:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13831.646378] LustreError: 429:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [13831.702884] Lustre: server umount lustre-OST0000 complete [13831.879516] LustreError: 22926:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13831.895363] LustreError: 22926:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [13834.918805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13834.939137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13834.955664] Lustre: Skipped 1 previous similar message [13834.961379] LustreError: 22926:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13841.167215] LustreError: 14011:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13841.180164] LustreError: 14011:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [13843.306694] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13843.320211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13843.403313] Lustre: 1605:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [13843.554059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13843.569582] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13844.683241] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13844.714136] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13844.719180] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [13844.722961] Lustre: Skipped 3 previous similar messages [13846.535849] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [13849.384344] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 22:45:20 (1746153920) [13877.263285] Lustre: Failing over lustre-OST0000 [13877.287998] LustreError: 3012:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13877.298006] LustreError: 3012:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13877.334044] Lustre: server umount lustre-OST0000 complete [13878.599295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13878.606313] Lustre: Skipped 1 previous similar message [13878.609836] LustreError: 13494:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13878.630731] LustreError: 13494:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [13882.430823] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13882.445623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13882.576968] Lustre: 3609:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [13882.741676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13882.758639] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13884.294231] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13884.806846] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [13884.807495] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13884.832980] Lustre: Skipped 1 previous similar message [13886.397819] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [13889.369057] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 22:46:00 (1746153960) [13908.930316] Lustre: Failing over lustre-OST0000 [13908.973730] LustreError: 5005:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13908.976944] LustreError: 5005:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13909.009907] Lustre: server umount lustre-OST0000 complete [13909.860776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13909.877995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13909.899579] Lustre: Skipped 1 previous similar message [13909.910580] LustreError: 22924:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13909.939454] LustreError: 22924:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [13915.050180] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13915.063023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13915.155528] Lustre: 5601:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [13915.284809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13915.298631] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13916.970469] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13917.048241] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [13917.051740] Lustre: Skipped 1 previous similar message [13917.057433] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13918.225813] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [13920.494896] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 22:46:32 (1746153992) [13952.886805] Lustre: Failing over lustre-OST0000 [13952.911840] LustreError: 8042:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13952.915899] LustreError: 8042:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13952.940367] Lustre: server umount lustre-OST0000 complete [13955.337374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13955.346249] Lustre: Skipped 1 previous similar message [13955.352955] LustreError: 13495:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13955.375261] LustreError: 13495:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [13957.785005] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13957.795096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13957.864996] Lustre: 8639:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [13957.990391] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13958.010827] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13959.255547] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13959.593626] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [13959.594046] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13959.612001] Lustre: Skipped 1 previous similar message [13960.727429] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [13962.862568] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 22:47:14 (1746154034) [13978.623894] Lustre: Failing over lustre-OST0000 [13978.656001] LustreError: 10083:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13978.660574] LustreError: 10083:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13978.694176] Lustre: server umount lustre-OST0000 complete [13979.655008] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13979.656532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13979.656533] Lustre: Skipped 1 previous similar message [13979.678492] LustreError: Skipped 2 previous similar messages [13984.322507] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [13984.342732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13984.485560] Lustre: 10680:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [13984.703820] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13984.733920] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13986.120402] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13986.605272] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [13986.610043] Lustre: Skipped 1 previous similar message [13986.613159] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13987.925340] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [13990.494587] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 22:47:42 (1746154062) [13997.223307] Lustre: Failing over lustre-OST0000 [13997.259998] LustreError: 11524:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [13997.273478] LustreError: 11524:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13997.340104] Lustre: server umount lustre-OST0000 complete [14009.755159] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14009.765527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14009.834620] Lustre: 12606:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [14009.983592] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [14010.004040] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [14011.422457] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [14011.833033] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [14011.836678] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [14011.836680] Lustre: Skipped 1 previous similar message [14013.131462] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [14015.632366] Lustre: DEBUG MARKER: == sanity test 165g: ofd_access_log_reader --keepalive works ========================================================== 22:48:07 (1746154087) [14058.101650] Lustre: Failing over lustre-OST0000 [14058.133447] LustreError: 13554:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [14058.147389] LustreError: 13554:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [14058.183195] Lustre: server umount lustre-OST0000 complete [14060.091898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14060.098838] Lustre: Skipped 3 previous similar messages [14060.102480] LustreError: 13494:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14060.141154] LustreError: 13494:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 15 previous similar messages [14071.646710] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [14071.665652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14071.764379] Lustre: 14704:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [14071.902611] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [14071.913742] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [14073.077727] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [14073.107627] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [14073.107633] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [14073.129977] Lustre: Skipped 1 previous similar message [14075.511276] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [14078.362227] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 22:49:09 (1746154149) [14079.423446] Lustre: DEBUG MARKER: creating a 10 Mb file [14121.806541] Lustre: DEBUG MARKER: starting reads [14123.174622] Lustre: DEBUG MARKER: truncating the file [14124.611027] Lustre: DEBUG MARKER: killing dd [14125.886838] Lustre: DEBUG MARKER: removing the temporary file [14131.088464] Lustre: DEBUG MARKER: == sanity test 170a: test lctl df to handle corrupted log ========================================================== 22:50:02 (1746154202) [14137.523856] Lustre: DEBUG MARKER: == sanity test 170b: check filename encoding ============= 22:50:09 (1746154209) [14157.914359] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 22:50:29 (1746154229) [14166.730165] Lustre: DEBUG MARKER: == sanity test 172: manual device removal with lctl cleanup/detach ================================================================ 22:50:38 (1746154238) [14173.486685] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 22:50:45 (1746154245) [14174.718204] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [14176.490891] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 22:50:47 (1746154247) [14178.258590] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_module obdecho/obdecho [14192.243521] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 22:51:03 (1746154263) [14194.142114] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_module obdecho/obdecho [14194.258504] Lustre: Echo OBD driver; http://www.lustre.org/ [14209.830841] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 22:51:21 (1746154281) [14280.969137] Lustre: DEBUG MARKER: == sanity test 182a: Test parallel modify metadata operations from mdc ========================================================== 22:52:32 (1746154352) [14325.881454] Lustre: DEBUG MARKER: == sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 22:53:17 (1746154397) [14865.247575] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 23:02:16 (1746154936) [14866.082773] Lustre: *** cfs_fail_loc=148, val=0*** [14870.499478] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 23:02:22 (1746154942) [14877.279816] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 23:02:29 (1746154949) [14882.920019] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 23:02:34 (1746154954) [14913.904990] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 23:03:05 (1746154985) [14921.582989] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 23:03:13 (1746154993) [14929.538721] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 23:03:20 (1746155000) [14934.125114] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 23:03:25 (1746155005) [14940.068857] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 23:03:31 (1746155011) [14950.316831] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 23:03:41 (1746155021) [14958.038001] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 23:03:49 (1746155029) [14963.011739] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 23:03:54 (1746155034) [14989.182164] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 23:04:20 (1746155060) [14994.326966] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 23:04:25 (1746155065) [14998.986829] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 23:04:30 (1746155070) [15003.736877] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 23:04:35 (1746155075) [15008.167671] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 23:04:39 (1746155079) [15012.784741] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 23:04:44 (1746155084) [15017.100699] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 23:04:48 (1746155088) [15021.818415] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 23:04:53 (1746155093) [15026.779977] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 23:04:58 (1746155098) [15031.890177] Lustre: lustre-MDD0000: changelog on [15031.892409] Lustre: Skipped 5 previous similar messages [15035.865713] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [15036.912275] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.9405 [15038.513538] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [15039.465693] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.12131 [15041.211873] Lustre: DEBUG MARKER: Test: lfs mkdir -i 1 /mnt/lustre/d205a.sanity.remote [15042.356201] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.12954 [15043.972517] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [15044.760730] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.12519 [15046.117364] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [15046.918686] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.4426 [15048.229722] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [15049.220746] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.4858 [15050.764413] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [15051.614198] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.32060 [15053.828959] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [15054.687378] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.29169 [15056.485145] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [15057.552928] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.9664 [15059.497544] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [15060.515762] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.6484 [15063.046571] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [15064.094949] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.24363 [15066.121743] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [15067.425944] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.11315 [15074.480019] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [15075.718295] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg126-client.v [15077.922039] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [15078.911831] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg126-client.E [15080.686889] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [15081.978602] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.oleg126-client.v [15088.306224] Lustre: lustre-MDD0001: changelog off [15088.308534] Lustre: Skipped 7 previous similar messages [15095.027389] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 23:06:06 (1746155166) [15103.963346] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 23:06:15 (1746155175) [15109.600955] Lustre: DEBUG MARKER: == sanity test 205d: verify the format of some stats files ========================================================== 23:06:21 (1746155181) [15123.590010] Lustre: DEBUG MARKER: == sanity test 205e: verify the output of lljobstat ====== 23:06:34 (1746155194) [15138.640490] Lustre: DEBUG MARKER: == sanity test 205f: verify qos_ost_weights YAML format == 23:06:49 (1746155209) [15147.038129] Lustre: DEBUG MARKER: == sanity test 205g: stress test for job_stats procfile == 23:06:58 (1746155218) [15245.830526] Lustre: DEBUG MARKER: == sanity test 205h: check jobid xattr is stored correctly ========================================================== 23:08:37 (1746155317) [15256.001883] Lustre: DEBUG MARKER: == sanity test 205i: check job_xattr parameter accepts and rejects values correctly ========================================================== 23:08:47 (1746155327) [15267.671628] Lustre: DEBUG MARKER: == sanity test 205k: Verify '?' operator on job stats ==== 23:08:59 (1746155339) [15276.179521] Lustre: DEBUG MARKER: == sanity test 205l: Verify job stats can scale ========== 23:09:07 (1746155347) [15379.475434] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 23:10:50 (1746155450) [15383.978800] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 23:10:55 (1746155455) [15388.755288] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 23:11:00 (1746155460) [15394.311410] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 23:11:05 (1746155465) [15402.130704] Lustre: Failing over lustre-MDT0000 [15402.383057] LustreError: 13276:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [15402.387094] LustreError: 13276:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [15402.550597] Lustre: server umount lustre-MDT0000 complete [15404.408274] LustreError: 2009:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15404.419478] LustreError: 2009:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [15404.532632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15404.537696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15404.543946] Lustre: Skipped 1 previous similar message [15418.768215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15418.941793] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15419.270592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15419.352300] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [15419.405678] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [15422.412767] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [15424.280859] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [15424.290694] Lustre: Skipped 1 previous similar message [15424.311558] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [15424.361885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:36076 to 0x280000403:36097) [15424.362501] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:36897 to 0x2c0000403:36929) [15428.277387] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15430.033743] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15435.260014] Lustre: Failing over lustre-MDT0000 [15435.419627] LustreError: 14591:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [15435.423058] LustreError: 14591:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [15435.513291] Lustre: server umount lustre-MDT0000 complete [15439.321920] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15439.323219] LustreError: 2008:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15439.323221] LustreError: 2008:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 15 previous similar messages [15439.346408] Lustre: Skipped 6 previous similar messages [15450.953710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15451.043630] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15451.260393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15451.293661] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [15453.332365] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [15454.482027] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [15456.269574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [15456.274343] Lustre: Skipped 3 previous similar messages [15456.302837] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [15456.331874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:36076 to 0x280000403:36129) [15456.334666] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:36897 to 0x2c0000403:36961) [15458.727985] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15459.919663] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15466.857240] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 23:12:18 (1746155538) [15485.449621] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 23:12:36 (1746155556) [15492.932394] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 23:12:44 (1746155564) [15501.797070] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 23:12:53 (1746155573) [15517.449585] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 23:13:09 (1746155589) [15549.598015] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 23:13:40 (1746155620) [15555.666274] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 23:13:47 (1746155627) [15572.852452] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with embedded hyphen ('-') ========================================================== 23:14:04 (1746155644) [15581.091865] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 23:14:12 (1746155652) [15582.471014] Lustre: DEBUG MARKER: creating a 10 Mb file [15640.116787] Lustre: DEBUG MARKER: starting reads [15641.390372] Lustre: DEBUG MARKER: truncating the file [15642.972239] Lustre: DEBUG MARKER: killing dd [15644.222544] Lustre: DEBUG MARKER: removing the temporary file [15651.009760] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 23:15:22 (1746155722) [15658.023504] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 23:15:29 (1746155729) [15661.699897] Lustre: *** cfs_fail_loc=229, val=0*** [15663.814092] Lustre: *** cfs_fail_loc=229, val=1*** [15663.819660] Lustre: Skipped 2 previous similar messages [15666.707647] Lustre: *** cfs_fail_loc=229, val=0*** [15668.836105] Lustre: *** cfs_fail_loc=229, val=1*** [15668.838380] Lustre: Skipped 4 previous similar messages [15681.733414] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 23:15:53 (1746155753) [15690.614295] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 23:16:01 (1746155761) [15696.706895] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 23:16:08 (1746155768) [15702.835178] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 23:16:14 (1746155774) [15709.688623] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 23:16:21 (1746155781) [15709.890070] Lustre: lustre-OST0001: Client a1856cf1-9991-42eb-b4d8-4abf2e431133 (at 192.168.201.26@tcp) reconnecting [15715.768276] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 23:16:27 (1746155787) [15719.306606] LustreError: 26479:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 sleeping for 3000ms [15722.330412] LustreError: 26479:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 awake [15730.561045] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 23:16:41 (1746155801) [15740.716217] Lustre: *** cfs_fail_loc=520, val=57344*** [15740.767523] LNet: *** cfs_fail_loc=e000, val=2147483648*** [15745.988655] Lustre: lustre-OST0000: Client a1856cf1-9991-42eb-b4d8-4abf2e431133 (at 192.168.201.26@tcp) reconnecting [15757.884631] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 23:17:09 (1746155829) [15761.249677] LustreError: 13499:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 sleeping for 22000ms [15782.139587] Lustre: lustre-OST0000: Client a1856cf1-9991-42eb-b4d8-4abf2e431133 (at 192.168.201.26@tcp) reconnecting [15783.288033] LustreError: 13499:0:(tgt_handler.c:2558:tgt_brw_read()) cfs_fail_timeout id 515 awake [15783.298958] LustreError: 13499:0:(ldlm_lib.c:3490:target_bulk_io()) @@@ bulk READ failed: rc = -107 req@ffff88008ac5ad80 x1830970271622016/t0(0) o3->a1856cf1-9991-42eb-b4d8-4abf2e431133@192.168.201.26@tcp:0/0 lens 488/440 e 0 to 0 dl 1746155854 ref 1 fl Interpret:/200/0 rc 0/0 job:'dd.0' uid:0 gid:0 [15783.330534] Lustre: 13499:0:(service.c:2494:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/2s); client may timeout req@ffff88008ac5ad80 x1830970271622016/t0(0) o3->a1856cf1-9991-42eb-b4d8-4abf2e431133@192.168.201.26@tcp:0/0 lens 488/440 e 0 to 0 dl 1746155854 ref 1 fl Complete:/200/0 rc 0/0 job:'dd.0' uid:0 gid:0 [15790.009292] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [15791.348130] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [15792.540359] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 23:17:44 (1746155864) [15802.246569] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 23:17:54 (1746155874) [15807.012578] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 23:17:58 (1746155878) [15813.259049] Lustre: DEBUG MARKER: == sanity test 226d: verify fid2path with -n and -fn option ========================================================== 23:18:04 (1746155884) [15819.447550] Lustre: DEBUG MARKER: == sanity test 226e: Verify path2fid -0 option with newline and space ========================================================== 23:18:11 (1746155891) [15825.611075] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 23:18:16 (1746155896) [15830.977787] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 23:18:22 (1746155902) [16020.253571] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 23:21:31 (1746156091) [16164.726001] Lustre: Failing over lustre-MDT0000 [16167.076611] LustreError: 28782:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [16167.080924] LustreError: 28782:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [16167.148272] Lustre: server umount lustre-MDT0000 complete [16167.517539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16167.518852] LustreError: 26293:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16167.518854] LustreError: 26293:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 14 previous similar messages [16167.551177] Lustre: Skipped 3 previous similar messages [16172.518631] LustreError: 7249:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16172.535252] LustreError: 7249:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 7 previous similar messages [16172.556707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16172.687397] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16172.986039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16173.039034] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [16173.617772] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [16175.224497] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [16177.976440] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [16177.986339] Lustre: Skipped 3 previous similar messages [16178.004169] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [16178.051018] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:43140 to 0x2c0000403:43169) [16178.051239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:42344 to 0x280000403:42369) [16206.499592] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 23:24:38 (1746156278) [16527.872837] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 23:29:59 (1746156599) [16534.130781] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 23:30:05 (1746156605) [16541.682010] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 23:30:12 (1746156612) [16595.570789] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 23:31:07 (1746156667) [16597.578629] Lustre: *** cfs_fail_loc=1801, val=0*** [16597.581438] LustreError: 2021:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240001b77:0x2f6e:0x0]/f1 failed: rc = -5 [16599.010468] LustreError: 2021:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0000: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush migrate_dir' to finish migration: rc = -1 [16599.026580] LustreError: 2021:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200004281:0x56fa:0x0]/migrate_dir failed: rc = -1 [16601.693250] Lustre: 13584:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0001: [0x240001b71:0x16:0x0]/foo is open, migrate only dentry [16608.865818] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [16610.520966] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 23:31:21 (1746156681) [16618.940082] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 23:31:30 (1746156690) [16626.671010] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 23:31:38 (1746156698) [16632.718685] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 23:31:44 (1746156704) [16639.658469] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 23:31:50 (1746156710) [16645.531730] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 23:31:57 (1746156717) [16651.787200] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 23:32:03 (1746156723) [16652.984779] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [16654.552435] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 23:32:06 (1746156726) [16757.956319] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 23:33:49 (1746156829) [16770.373399] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 23:34:01 (1746156841) [16777.721375] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 23:34:09 (1746156849) [16803.842360] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 23:34:34 (1746156874) [16835.176835] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 23:35:06 (1746156906) [16876.919607] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 23:35:48 (1746156948) [16885.312571] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 23:35:56 (1746156956) [16895.961996] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 23:36:07 (1746156967) [16904.225956] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 23:36:15 (1746156975) [16905.474857] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [16907.298988] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 23:36:18 (1746156978) [16908.646571] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [16910.513675] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 23:36:21 (1746156981) [16919.811044] Lustre: DEBUG MARKER: == sanity test 230x: dir migration check space =========== 23:36:31 (1746156991) [16930.257834] LustreError: 2021:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000007:0x1:0x0]/d230x.sanity failed: rc = -28 [16930.264953] LustreError: 2021:0:(mdt_reint.c:2523:mdt_reint_migrate()) Skipped 1 previous similar message [16974.631341] Lustre: DEBUG MARKER: == sanity test 230y: unlink dir with bad hash type ======= 23:37:26 (1746157046) [16977.752589] Lustre: *** cfs_fail_loc=1802, val=0*** [16977.875108] LustreError: 13584:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240001b77:0x35ce:0x0]/d59 failed: rc = -77 [16992.992562] Lustre: DEBUG MARKER: == sanity test 230z: resume dir migration with bad hash type ========================================================== 23:37:44 (1746157064) [16996.328708] Lustre: *** cfs_fail_loc=1802, val=0*** [16996.333301] Lustre: Skipped 1 previous similar message [16996.502324] LustreError: 13584:0:(mdt_reint.c:2523:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240001b77:0x3605:0x0]/d50 failed: rc = -77 [17045.632656] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 23:38:36 (1746157116) [17054.362193] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 23:38:45 (1746157125) [17105.316488] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 23:39:36 (1746157176) [17106.300143] Lustre: *** cfs_fail_loc=31c, val=0*** [17110.961001] Lustre: Failing over lustre-OST0000 [17111.165614] LustreError: 10045:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [17111.177464] LustreError: 10045:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [17111.343718] Lustre: server umount lustre-OST0000 complete [17114.237524] LustreError: 22931:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [17114.537400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17114.550634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17118.148119] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [17118.164237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [17118.268557] Lustre: 10643:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [17118.436291] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [17118.459477] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [17119.211416] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [17120.351424] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [17120.358747] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [17120.358749] Lustre: Skipped 3 previous similar messages [17122.714170] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [17129.129824] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 23:40:00 (1746157200) [17130.249629] Lustre: *** cfs_fail_loc=31c, val=0*** [17130.257116] LustreError: 4812:0:(ldlm_request.c:475:ldlm_cli_enqueue_local()) ### delayed lvb init failed (rc -12) ns: filter-lustre-OST0000_UUID lock: ffff880098f04000/0x8dc667129bce8d7a lrc: 2/0,0 mode: --/PR res: [0x280000403:0xd2aa:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 4812 timeout: 0 lvb_type: 0 [17133.757124] Lustre: Failing over lustre-OST0000 [17133.801170] LustreError: 11706:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [17133.817700] LustreError: 11706:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [17133.891572] Lustre: server umount lustre-OST0000 complete [17135.401261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17135.418344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17135.424726] LustreError: 22924:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17135.424728] LustreError: 22924:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [17135.485170] Lustre: Skipped 2 previous similar messages [17138.457238] LustreError: 22924:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17138.469851] LustreError: 22924:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [17140.233169] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [17140.246544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [17140.311537] Lustre: 12305:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [17140.487400] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [17140.499965] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [17141.613116] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [17141.792649] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [17141.799098] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [17141.799100] Lustre: Skipped 1 previous similar message [17144.164638] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [17150.798811] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 23:40:22 (1746157222) [17157.428785] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 23:40:28 (1746157228) [17163.978451] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 23:40:35 (1746157235) [17169.924265] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 23:40:41 (1746157241) [17177.722707] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 23:40:48 (1746157248) [17184.569594] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 23:40:56 (1746157256) [17190.109816] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 23:41:01 (1746157261) [17281.377594] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 23:42:32 (1746157352) [17284.050811] Lustre: *** cfs_fail_loc=2100, val=0*** [17290.712256] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 23:42:42 (1746157362) [17300.187340] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 23:42:51 (1746157371) [17302.550545] Lustre: *** cfs_fail_loc=713, val=1*** [17302.553647] Lustre: Skipped 1 previous similar message [17312.067369] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 23:43:03 (1746157383) [17392.301790] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 23:44:23 (1746157463) [17425.161876] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 23:44:56 (1746157496) [17426.227024] Lustre: *** cfs_fail_loc=105, val=0*** [17431.641243] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 23:45:03 (1746157503) [17712.407955] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 23:49:43 (1746157783) [17826.200174] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 23:51:37 (1746157897) [17831.847117] Lustre: DEBUG MARKER: == sanity test 245a: check mdc connection flag/data: multiple modify RPCs ========================================================== 23:51:43 (1746157903) [17836.632511] Lustre: DEBUG MARKER: == sanity test 245b: check osp connection flag/data: multiple modify RPCs ========================================================== 23:51:48 (1746157908) [17843.647004] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 23:51:55 (1746157915) [17850.771009] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 23:52:02 (1746157922) [17857.131628] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 23:52:08 (1746157928) [17863.770438] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 23:52:15 (1746157935) [17871.269985] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 23:52:22 (1746157942) [17878.048623] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 23:52:29 (1746157949) [17879.492712] Lustre: enable_remote_subdir_mount is deprecated, it's always enabled. [17894.330732] Lustre: DEBUG MARKER: == sanity test 247g: striped directory submount revalidate ROOT from cache ========================================================== 23:52:46 (1746157966) [17902.676117] Lustre: DEBUG MARKER: == sanity test 247h: remote directory submount revalidate ROOT from cache ========================================================== 23:52:54 (1746157974) [17913.251602] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 23:53:04 (1746157984) [18008.207722] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 23:54:39 (1746158079) [18047.980898] Lustre: DEBUG MARKER: == sanity test 248c: verify whole file read behavior ===== 23:55:19 (1746158119) [18064.736596] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 23:55:36 (1746158136) [18069.919314] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 23:55:41 (1746158141) [18075.377321] Lustre: DEBUG MARKER: == sanity test 251a: Handling short read and write correctly ========================================================== 23:55:47 (1746158147) [18082.116165] Lustre: DEBUG MARKER: == sanity test 251b: short read restore offset correctly ========================================================== 23:55:53 (1746158153) [18092.511505] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 23:56:04 (1746158164) [18101.878287] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 23:56:13 (1746158173) [18140.961613] LustreError: 7249:0:(lod_qos.c:1380:lod_ost_alloc_specific()) can't lstripe objid [0x200004284:0x42:0x0]: have 0 want 1 [18185.338162] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 23:57:37 (1746158257) [18187.880092] Lustre: lustre-MDD0000: changelog on [18187.882419] Lustre: Skipped 1 previous similar message [18195.621409] Lustre: lustre-MDD0001: changelog off [18195.626261] Lustre: Skipped 1 previous similar message [18199.060561] Lustre: DEBUG MARKER: SKIP: sanity test_255a skipping excluded test 255a (base 255) [18200.664353] Lustre: DEBUG MARKER: SKIP: sanity test_255b skipping excluded test 255b (base 255) [18202.489462] Lustre: DEBUG MARKER: SKIP: sanity test_255c skipping excluded test 255c (base 255) [18204.144672] Lustre: DEBUG MARKER: SKIP: sanity test_256 skipping excluded test 256 [18205.390458] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 23:57:57 (1746158277) [18206.671302] Lustre: *** cfs_fail_loc=161, val=0*** [18207.772531] Lustre: Failing over lustre-MDT0001 [18211.692573] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.26@tcp (stopping) [18211.708248] Lustre: Skipped 1 previous similar message [18212.244816] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [18212.249679] LustreError: Skipped 1 previous similar message [18212.251901] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18212.283508] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [18214.171509] LustreError: 29132:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [18214.174929] LustreError: 29132:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [18214.351052] Lustre: server umount lustre-MDT0001 complete [18216.672587] LustreError: 24737:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [18217.565883] LustreError: 32545:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18217.579710] LustreError: 32545:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [18220.632504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18220.733132] Lustre: 29751:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [18220.958610] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [18220.989430] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [18221.666447] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [18222.914422] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18225.393580] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [18225.959464] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [18225.963405] Lustre: Skipped 1 previous similar message [18225.972160] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [18225.996381] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:22563 to 0x2c0000400:22593) [18225.996930] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:22779 to 0x280000400:22817) [18234.883977] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 23:58:26 (1746158306) [18239.633634] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 23:58:31 (1746158311) [18244.582472] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 23:58:36 (1746158316) [18252.233644] Lustre: *** cfs_fail_loc=2301, val=0*** [18253.931011] Lustre: Failing over lustre-OST0000 [18253.965487] LustreError: 31905:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [18253.969153] LustreError: 31905:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [18254.006385] Lustre: server umount lustre-OST0000 complete [18256.021625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18256.026848] LustreError: 4809:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18256.045841] Lustre: Skipped 3 previous similar messages [18259.936820] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18259.956156] LDISKFS-fs (dm-2): 1 truncate cleaned up [18259.960879] LDISKFS-fs (dm-2): recovery complete [18259.970512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18260.052496] Lustre: 32567:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [18260.347722] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [18260.361704] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [18261.473455] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [18262.313635] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [18262.314291] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [18262.326848] Lustre: Skipped 2 previous similar messages [18263.499660] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18272.121417] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 23:59:03 (1746158343) [18276.849492] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 23:59:08 (1746158348) [18287.806427] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 23:59:19 (1746158359) [18293.203102] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 23:59:24 (1746158364) [18298.701121] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 23:59:30 (1746158370) [18304.188560] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 23:59:35 (1746158375) [18312.332247] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 23:59:43 (1746158383) [18317.045711] Lustre: Increasing provided stripe size to a minimum value 64 [18326.000174] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 23:59:57 (1746158397) [18331.223673] Lustre: *** cfs_fail_loc=168, val=40*** [18332.837936] Lustre: *** cfs_fail_loc=168, val=20*** [18332.843363] Lustre: Skipped 2 previous similar messages [18334.198038] Lustre: *** cfs_fail_loc=168, val=0*** [18334.203314] Lustre: Skipped 2 previous similar messages [18337.022761] Lustre: *** cfs_fail_loc=168, val=30*** [18337.025119] Lustre: Skipped 5 previous similar messages [18345.180958] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 00:00:16 (1746158416) [18352.664065] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 00:00:24 (1746158424) [18358.325402] Lustre: DEBUG MARKER: == sanity test 270j: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 00:00:29 (1746158429) [18364.088813] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 00:00:35 (1746158435) [18369.045125] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 00:00:40 (1746158440) [18374.656704] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 00:00:46 (1746158446) [18380.134768] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 00:00:51 (1746158451) [18484.300388] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 00:02:36 (1746158556) [18489.560690] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 00:02:41 (1746158561) [18494.900274] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 00:02:46 (1746158566) [18501.463883] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 00:02:52 (1746158572) [18508.041395] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 00:02:59 (1746158579) [18517.019075] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 00:03:08 (1746158588) [18525.453206] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 00:03:16 (1746158596) [18534.371353] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 00:03:25 (1746158605) [18542.658123] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 00:03:34 (1746158614) [18549.855775] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 00:03:41 (1746158621) [18556.652716] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 00:03:48 (1746158628) [18558.622869] LustreError: 13584:0:(mdt_io.c:571:mdt_commitrw_write()) cfs_fail_timeout id 16b sleeping for 2000ms [18560.648955] LustreError: 13584:0:(mdt_io.c:571:mdt_commitrw_write()) cfs_fail_timeout id 16b awake [18566.281303] Lustre: DEBUG MARKER: == sanity test 273c: race writeback and object destroy === 00:03:57 (1746158637) [18574.799916] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 00:04:06 (1746158646) [18583.386636] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 00:04:14 (1746158654) [18584.883343] Lustre: Failing over lustre-OST0000 [18584.948009] LustreError: 12959:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [18584.954046] LustreError: 12959:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [18584.985553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18585.004043] LustreError: 4811:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18585.013334] LustreError: 4811:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [18585.067061] Lustre: server umount lustre-OST0000 complete [18589.992799] LustreError: 4813:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18590.026153] LustreError: 4813:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [18591.645660] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18591.658645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18591.764340] Lustre: 14147:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [18591.827752] LustreError: 14175:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [18592.030346] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [18592.043610] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [18592.413924] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [18593.896721] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [18593.898025] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [18593.898027] Lustre: Skipped 1 previous similar message [18596.004207] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18599.052558] Lustre: Failing over lustre-OST0000 [18599.062130] LustreError: 15422:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [18599.141680] LustreError: 15417:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [18599.144601] LustreError: 15417:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [18599.435713] Lustre: server umount lustre-OST0000 complete [18602.021942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18602.047147] Lustre: Skipped 1 previous similar message [18602.063532] LustreError: 4811:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18602.091064] LustreError: 4811:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [18606.171517] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18606.197616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18606.364238] Lustre: 16644:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [18606.417324] LustreError: 16684:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [18606.566226] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [18606.582839] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [18607.445943] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [18607.919593] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [18607.919606] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [18607.936103] Lustre: Skipped 1 previous similar message [18610.196844] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18612.941809] Lustre: Failing over lustre-OST0000 [18612.947827] LustreError: 17959:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [18612.948651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -19 [18612.948664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18612.948666] Lustre: Skipped 1 previous similar message [18613.024704] LustreError: 17954:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [18613.033947] LustreError: 17954:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [18613.166284] Lustre: server umount lustre-OST0000 complete [18618.414431] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18618.428628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18618.521642] Lustre: 19164:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [18618.561153] LustreError: 19194:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [18618.662957] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [18618.678219] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [18619.761041] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [18619.855348] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [18619.855357] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [18619.875519] Lustre: Skipped 1 previous similar message [18621.671779] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18624.053945] Lustre: Failing over lustre-OST0000 [18624.123320] LustreError: 20462:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [18624.128105] LustreError: 20462:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [18624.155155] Lustre: server umount lustre-OST0000 complete [18624.867595] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [18624.880709] LustreError: Skipped 1 previous similar message [18624.882992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18624.889327] LustreError: 22931:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18624.889329] LustreError: 22931:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [18624.911376] Lustre: Skipped 2 previous similar messages [18629.337274] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18629.346271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18629.770212] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [18629.791130] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [18631.791948] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [18631.896817] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [18631.902416] Lustre: Skipped 1 previous similar message [18631.904697] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [18633.320838] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18635.717697] Lustre: Failing over lustre-OST0000 [18635.866578] Lustre: server umount lustre-OST0000 complete [18636.948618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [18636.958366] LustreError: Skipped 1 previous similar message [18641.138975] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18641.154256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18641.268372] Lustre: 24215:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [18641.275266] Lustre: 24215:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message [18644.730524] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18647.373564] Lustre: Failing over lustre-OST0000 [18647.381227] LustreError: 25548:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [18647.385465] LustreError: 25548:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [18647.436109] LustreError: 25541:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [18647.439942] LustreError: 25541:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [18647.485818] Lustre: server umount lustre-OST0000 complete [18648.469175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [18648.484962] LustreError: Skipped 1 previous similar message [18648.487596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18648.502833] Lustre: Skipped 3 previous similar messages [18652.453824] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18652.464062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18652.716463] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [18652.726549] Lustre: Skipped 1 previous similar message [18652.751422] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [18652.772424] Lustre: Skipped 1 previous similar message [18654.044801] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [18654.055075] Lustre: Skipped 1 previous similar message [18654.116312] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.126@tcp (at 0@lo) [18654.118373] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [18654.118375] Lustre: Skipped 1 previous similar message [18654.133638] Lustre: Skipped 3 previous similar messages [18655.624454] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18657.605065] Lustre: Failing over lustre-OST0000 [18657.666296] Lustre: server umount lustre-OST0000 complete [18657.703197] LustreError: 14812:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18657.712900] LustreError: 14812:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 16 previous similar messages [18662.964287] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18662.980255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18666.171804] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18668.642154] Lustre: Failing over lustre-OST0000 [18668.658056] LustreError: 30550:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [18668.661537] LustreError: 30550:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [18668.832827] Lustre: server umount lustre-OST0000 complete [18670.022376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [18670.028854] LustreError: Skipped 2 previous similar messages [18674.375549] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18674.385927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18674.524867] Lustre: 31719:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [18674.534469] Lustre: 31719:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 2 previous similar messages [18678.152974] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18680.488956] Lustre: Failing over lustre-OST0000 [18680.556991] LustreError: 557:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [18680.559794] LustreError: 557:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [18680.627884] Lustre: server umount lustre-OST0000 complete [18681.878968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18681.889185] Lustre: Skipped 5 previous similar messages [18684.961495] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18684.972509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18685.187977] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [18685.194925] Lustre: Skipped 2 previous similar messages [18685.201746] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [18685.210105] Lustre: Skipped 2 previous similar messages [18686.674585] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [18686.680134] Lustre: Skipped 2 previous similar messages [18686.998480] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [18687.001252] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [18687.001254] Lustre: Skipped 5 previous similar messages [18687.029461] Lustre: Skipped 2 previous similar messages [18688.224531] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18690.519078] Lustre: Failing over lustre-OST0000 [18690.642137] Lustre: server umount lustre-OST0000 complete [18692.019754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [18692.027011] LustreError: Skipped 3 previous similar messages [18695.463958] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18695.486943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18698.794964] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18701.071830] Lustre: Failing over lustre-OST0000 [18701.076130] LustreError: 5608:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [18701.079373] LustreError: 5608:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [18701.237887] Lustre: server umount lustre-OST0000 complete [18707.904637] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18707.922271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18712.086401] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18714.627309] Lustre: Failing over lustre-OST0000 [18714.752786] Lustre: server umount lustre-OST0000 complete [18720.554629] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18720.570316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18724.631253] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18726.740611] Lustre: Failing over lustre-OST0000 [18726.853087] Lustre: server umount lustre-OST0000 complete [18727.844546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [18727.862973] LustreError: Skipped 4 previous similar messages [18727.869089] LustreError: 4811:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18727.869091] LustreError: 4811:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 29 previous similar messages [18732.461954] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18732.471653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18736.072396] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18737.838510] Lustre: Failing over lustre-OST0000 [18737.954292] Lustre: server umount lustre-OST0000 complete [18743.227660] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18743.237876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18743.325336] Lustre: 14295:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.201.126@tcp: no remote llog for lustre-sptlrpc, check MGS config [18743.331251] Lustre: 14295:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 5 previous similar messages [18746.235477] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18748.536168] Lustre: Failing over lustre-OST0000 [18748.620170] LustreError: 15587:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [18748.623971] LustreError: 15587:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [18748.657792] Lustre: server umount lustre-OST0000 complete [18750.180560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18750.188318] Lustre: Skipped 11 previous similar messages [18753.301030] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18753.315808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18753.663158] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [18753.668356] Lustre: Skipped 5 previous similar messages [18753.678371] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [18753.683118] Lustre: Skipped 5 previous similar messages [18754.775700] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [18754.780126] Lustre: Skipped 5 previous similar messages [18755.419568] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.126@tcp (at 0@lo) [18755.419761] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [18755.419762] Lustre: Skipped 5 previous similar messages [18755.429933] Lustre: Skipped 11 previous similar messages [18756.091070] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18757.716308] Lustre: Failing over lustre-OST0000 [18757.790855] Lustre: server umount lustre-OST0000 complete [18762.494455] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18762.507800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18765.480254] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18767.732790] Lustre: Failing over lustre-OST0000 [18767.733875] LustreError: 20543:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [18767.733877] LustreError: 20543:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [18767.770627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18767.776035] Lustre: Skipped 3 previous similar messages [18767.830689] Lustre: server umount lustre-OST0000 complete [18772.202515] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18772.223181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18775.692806] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18778.528042] Lustre: Failing over lustre-OST0000 [18778.720869] Lustre: server umount lustre-OST0000 complete [18784.402680] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18784.410033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18788.545043] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18791.068694] Lustre: Failing over lustre-OST0000 [18791.186067] Lustre: server umount lustre-OST0000 complete [18795.524413] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18795.534130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18798.321214] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18800.951743] Lustre: Failing over lustre-OST0000 [18801.104693] Lustre: server umount lustre-OST0000 complete [18802.134477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [18802.140137] LustreError: Skipped 8 previous similar messages [18806.375496] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [18806.387556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18809.418457] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18816.474939] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 00:08:08 (1746158888) [18821.409752] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 00:08:13 (1746158893) [18823.238005] Lustre: Failing over lustre-MDT0000 [18823.246597] Lustre: Failing over lustre-MDT0001 [18823.389746] LustreError: 31234:0:(tgt_mount.c:1756:server_put_super()) cfs_race id 60c sleeping [18823.680786] Lustre: server umount lustre-MDT0000 complete [18828.393027] LustreError: 31234:0:(tgt_mount.c:1756:server_put_super()) cfs_fail_race id 60c awake: rc=0 [18828.542305] Lustre: server umount lustre-MDT0001 complete [18828.785329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18828.905318] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18829.028392] LustreError: 31833:0:(tgt_mount.c:1486:server_start_targets()) cfs_fail_race id 60c waking [18831.237799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18833.289303] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18839.163520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [18839.527894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18841.625701] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18844.489648] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [18844.608631] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:22600 to 0x2c0000400:22625) [18844.610576] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:22833 to 0x280000400:22849) [18844.699089] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:54767 to 0x2c0000403:54785) [18844.699672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:53974 to 0x280000403:54017) [18854.064323] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 00:08:45 (1746158925) [18855.544568] LustreError: 31851:0:(llog_osd.c:1480:llog_osd_open()) cfs_race id 15e sleeping [18857.549537] Lustre: Failing over lustre-MDT0000 [18857.669726] LustreError: 1536:0:(tgt_mount.c:128:server_deregister_mount()) cfs_fail_race id 15e waking [18857.673359] LustreError: 31851:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e awake: rc=2875 [18858.695852] LustreError: 1536:0:(tgt_mount.c:128:server_deregister_mount()) cfs_fail_race id 15e waking [18858.724473] Lustre: server umount lustre-MDT0000 complete [18859.561318] LustreError: 31859:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18859.573939] LustreError: 31859:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 48 previous similar messages [18864.288162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18864.418544] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18864.465400] LustreError: 2140:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [18865.568607] LustreError: 2140:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [18866.591589] LustreError: 2140:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [18868.608857] LustreError: 2140:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [18868.613057] LustreError: 2140:0:(llog_osd.c:1480:llog_osd_open()) Skipped 1 previous similar message [18872.658519] LustreError: 2139:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [18872.663073] LustreError: 2139:0:(llog_osd.c:1480:llog_osd_open()) Skipped 3 previous similar messages [18876.734399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18877.729862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:53974 to 0x280000403:54049) [18877.737588] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:54767 to 0x2c0000403:54817) [18878.882528] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug all all [18881.184416] LustreError: 2141:0:(llog_osd.c:1480:llog_osd_open()) cfs_fail_race id 15e waking [18881.188755] LustreError: 2141:0:(llog_osd.c:1480:llog_osd_open()) Skipped 4 previous similar messages [18891.764557] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 00:09:23 (1746158963) [18897.486653] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 00:09:29 (1746158969) [18923.874121] Lustre: DEBUG MARKER: == sanity test 300c: chown [19190.870351] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 00:14:22 (1746159262) [19199.215531] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ========================================================== 00:14:30 (1746159270) [19205.619347] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ========================================================== 00:14:37 (1746159277) [19212.482821] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ========================================================== 00:14:44 (1746159284) [19225.776870] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory ========================================================== 00:14:57 (1746159297) [19240.063501] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory ========================================================== 00:15:11 (1746159311) [19302.320742] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ============ 00:16:13 (1746159373) [19307.679896] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ======== 00:16:19 (1746159379) [19314.100610] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ========================================================== 00:16:25 (1746159385) [19320.464575] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS ========================================================== 00:16:32 (1746159392) [19321.451537] Lustre: DEBUG MARKER: SKIP: sanity test_300m Only for single MDT [19322.623339] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA ========================================================== 00:16:34 (1746159394) [19334.259745] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [19335.718472] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================================== 00:16:47 (1746159407) [19336.760215] Lustre: *** cfs_fail_loc=1704, val=0*** [19336.764625] Lustre: 27031:0:(out_handler.c:911:out_tx_end()) lustre-MDT0001-osd: error during execution of #8 from /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:670: rc = -28 [19336.775756] LustreError: 27031:0:(out_handler.c:921:out_tx_end()) lustre-MDT0001-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:425: rc = -524 [19341.737838] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ========================================================== 00:16:53 (1746159413) [19346.405308] Lustre: DEBUG MARKER: == sanity test 300r: test -1 striped directory =========== 00:16:58 (1746159418) [19351.882949] Lustre: DEBUG MARKER: == sanity test 300s: test lfs mkdir -c without -i ======== 00:17:03 (1746159423) [19357.728640] Lustre: DEBUG MARKER: == sanity test 300t: test max_mdt_stripecount ============ 00:17:09 (1746159429) [19366.016666] Lustre: DEBUG MARKER: == sanity test 300ua: basic overstriped dir sanity test == 00:17:17 (1746159437) [19373.743885] Lustre: DEBUG MARKER: == sanity test 300ub: test MDT overstriping interface [19380.351365] Lustre: DEBUG MARKER: == sanity test 300uc: test MDT overstriping as default [19385.704008] Lustre: DEBUG MARKER: == sanity test 300ud: dir split ========================== 00:17:37 (1746159457) [19489.775993] Lustre: DEBUG MARKER: == sanity test 300ue: dir merge ========================== 00:19:21 (1746159561) [19579.344517] Lustre: DEBUG MARKER: == sanity test 300uf: migrate with too many local locks == 00:20:50 (1746159650) [19588.393666] Lustre: DEBUG MARKER: == sanity test 300ug: migrate overstriped dirs =========== 00:20:59 (1746159659) [19589.678343] Lustre: 31879:0:(osd_handler.c:2068:osd_trans_start()) lustre-MDT0000: credits 3793 > trans_max 3200 [19589.682776] Lustre: 31879:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 6/24/0, destroy: 6/24/0 [19589.686463] Lustre: 31879:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 31/2640/0 [19589.694566] Lustre: 31879:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 0/0/0 [19589.703124] Lustre: 31879:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 47/938/0, delete: 32/50/0 [19589.707498] Lustre: 31879:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 30/30/0, ref_del: 28/28/0 [19589.726559] CPU: 1 PID: 31879 Comm: mdt_io00_002 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 [19589.734813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [19589.739581] Call Trace: [19589.740740] [] dump_stack+0x19/0x1b [19589.745907] [] osd_trans_start+0x683/0x6b0 [osd_ldiskfs] [19589.751332] [] ? __llog_ctxt_put+0xd3/0x140 [obdclass] [19589.755397] [] top_trans_start+0x351/0xab0 [ptlrpc] [19589.761398] [] lod_trans_start+0x7f/0x2e0 [lod] [19589.765450] [] mdd_trans_start+0x14/0x20 [mdd] [19589.769971] [] mdd_migrate_object+0x12ee/0x2470 [mdd] [19589.776154] [] ? _raw_spin_unlock+0xe/0x20 [19589.780160] [] ? __mdt_stripe_get+0xe1/0x4e0 [mdt] [19589.783819] [] mdd_migrate+0x28/0x30 [mdd] [19589.790476] [] mdt_reint_migrate+0x1b9f/0x24b0 [mdt] [19589.794178] [] mdt_reint_rec+0x87/0x240 [mdt] [19589.798150] [] mdt_reint_internal+0x84f/0x13d0 [mdt] [19589.801715] [] ? mdt_thread_info_init+0xa5/0xc0 [mdt] [19589.807294] [] mdt_reint+0x67/0x150 [mdt] [19589.812568] [] tgt_request_handle+0x74e/0x1a60 [ptlrpc] [19589.818814] [] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc] [19589.824316] [] ptlrpc_main+0xc61/0x1640 [ptlrpc] [19589.829701] [] ? put_prev_entity+0x31/0x400 [19589.832575] [] ? do_raw_spin_unlock+0x49/0x90 [19589.836730] [] ? _raw_spin_unlock_irq+0xe/0x30 [19589.843402] [] ? ptlrpc_wait_event+0x630/0x630 [ptlrpc] [19589.849489] [] kthread+0xe4/0xf0 [19589.852889] [] ? kthread_create_on_node+0x140/0x140 [19589.859110] [] ret_from_fork_nospec_begin+0x7/0x21 [19589.865782] [] ? kthread_create_on_node+0x140/0x140 [19597.367405] Lustre: DEBUG MARKER: == sanity test 300uh: overstripe tunable max_stripes_per_mdt ========================================================== 00:21:08 (1746159668) [19606.744775] Lustre: DEBUG MARKER: == sanity test 300ui: overstripe is not supported on one MDT system ========================================================== 00:21:18 (1746159678) [19607.856721] Lustre: DEBUG MARKER: SKIP: sanity test_300ui 1 MDT only [19609.402047] Lustre: DEBUG MARKER: == sanity test 300uj: overstriped dir with -C -N sanity test ========================================================== 00:21:20 (1746159680) [19615.496265] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ============= 00:21:27 (1746159687) [19621.634668] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open ========================================================== 00:21:33 (1746159693) [19627.561503] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ========================================================== 00:21:39 (1746159699) [19628.836763] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [19630.123123] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ========================================================== 00:21:41 (1746159701) [19664.248462] Lustre: DEBUG MARKER: == sanity test 312: make sure ZFS adjusts its block size by write pattern ========================================================== 00:22:16 (1746159736) [19665.786361] Lustre: DEBUG MARKER: SKIP: sanity test_312 the test only applies to zfs [19667.096573] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ========================================================== 00:22:18 (1746159738) [19668.039606] Lustre: *** cfs_fail_loc=720, val=0*** [19668.043066] LustreError: 26478:0:(osd_handler.c:2213:osd_trans_stop()) lustre-OST0000: failed in transaction hook: rc = -5 [19668.048338] LustreError: 26478:0:(ofd_objects.c:1014:ofd_object_punch()) lustre-OST0000: failed to stop transaction: rc = -5 [19672.929297] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ========================================================== 00:22:24 (1746159744) [19675.099659] Lustre: *** cfs_fail_loc=720, val=0*** [19675.100619] LustreError: 4812:0:(osd_handler.c:2213:osd_trans_stop()) lustre-OST0001: failed in transaction hook: rc = -5 [19675.100638] LustreError: 4812:0:(ofd_objects.c:1081:ofd_destroy()) lustre-OST0001 failed to stop transaction: -5 [19675.100747] LustreError: 4812:0:(ofd_dev.c:1765:ofd_destroy_hdl()) lustre-OST0001: error destroying object [0x2c0000403:0xdb52:0x0]: -5 [19675.142970] Lustre: Skipped 1 previous similar message [19679.522096] LustreError: 2152:0:(osp_sync.c:1155:osp_sync_process_committed()) lustre-OST0001-osc-MDT0000: can't cancel record: rc = -2 [19682.280368] LustreError: 2150:0:(osp_sync.c:1155:osp_sync_process_committed()) lustre-OST0000-osc-MDT0000: can't cancel record: rc = -2 [19688.740717] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ============= 00:22:40 (1746159760) [19699.375226] Lustre: DEBUG MARKER: == sanity test 316: lfs migrate of file with large_xattr enabled ========================================================== 00:22:51 (1746159771) [19705.028475] Lustre: DEBUG MARKER: == sanity test 317: Verify blocks get correctly update after truncate ========================================================== 00:22:56 (1746159776) [19710.973050] Lustre: DEBUG MARKER: == sanity test 318: Verify async readahead tunables ====== 00:23:02 (1746159782) [19715.820898] Lustre: DEBUG MARKER: == sanity test 319: lost lease lock on migrate error ===== 00:23:07 (1746159787) [19721.317238] Lustre: 31879:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0000: [0x200005224:0x142:0x0]/xxx is open, migrate only dentry [19727.021872] Lustre: DEBUG MARKER: == sanity test 350: force NID mismatch path to be exercised ========================================================== 00:23:18 (1746159798) [19728.158694] LNet: *** cfs_fail_loc=e001, val=100*** [19728.161465] LNet: 31867:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-0@lo matched 1830970216676608 with NID mismatch 0@lo accepted (same peer ffff8800a8346e00) [19728.663351] LNet: *** cfs_fail_loc=e001, val=100*** [19728.675435] LNet: Skipped 28 previous similar messages [19728.677687] LNet: 9483:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-192.168.201.26@tcp matched 1830970216680704 with NID mismatch 192.168.201.26@tcp accepted (same peer ffff88009a261c00) [19728.698676] LNet: 9483:0:(lib-ptl.c:160:lnet_try_match_md()) Skipped 28 previous similar messages [19729.691101] LNet: *** cfs_fail_loc=e001, val=100*** [19729.697029] LNet: Skipped 42 previous similar messages [19729.699858] LNet: 9484:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-192.168.201.26@tcp matched 1830970216686464 with NID mismatch 192.168.201.26@tcp accepted (same peer ffff88009a261c00) [19729.714369] LNet: 9484:0:(lib-ptl.c:160:lnet_try_match_md()) Skipped 42 previous similar messages [19986.042431] Lustre: DEBUG MARKER: == sanity test 360: ldiskfs unlink in a separate thread == 00:27:37 (1746160057) [20006.898657] Lustre: DEBUG MARKER: == sanity test 398a: direct IO should cancel lock otherwise lockless ========================================================== 00:27:58 (1746160078) [20012.097986] Lustre: DEBUG MARKER: == sanity test 398b: DIO and buffer IO race ============== 00:28:03 (1746160083) [20148.755681] Lustre: DEBUG MARKER: == sanity test 398c: run fio to test AIO ================= 00:30:20 (1746160220) [20199.466044] Lustre: DEBUG MARKER: == sanity test 398d: run aiocp to verify block size > stripe size ========================================================== 00:31:11 (1746160271) [20224.609338] Lustre: DEBUG MARKER: == sanity test 398e: O_Direct open cleared by fcntl doesn't cause hang ========================================================== 00:31:36 (1746160296) [20229.967328] Lustre: DEBUG MARKER: == sanity test 398f: verify aio handles ll_direct_rw_pages errors correctly ========================================================== 00:31:41 (1746160301) [20240.473738] Lustre: DEBUG MARKER: == sanity test 398g: verify parallel dio async RPC submission ========================================================== 00:31:52 (1746160312) [20241.885112] LustreError: 19848:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [20241.911733] LustreError: 19848:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 7 previous similar messages [20243.895016] LustreError: 13501:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [20243.909831] LustreError: 13501:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 6 previous similar messages [20244.429133] LustreError: 26478:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [20246.442970] LustreError: 26478:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [20246.451295] LustreError: 26478:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [20246.507821] LustreError: 26478:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [20248.524030] LustreError: 26478:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [20248.581092] LustreError: 13501:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [20250.588082] LustreError: 13501:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [20252.703988] LustreError: 13501:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 sleeping for 2000ms [20252.711108] LustreError: 13501:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [20254.716201] LustreError: 13501:0:(tgt_handler.c:2704:tgt_brw_write()) cfs_fail_timeout id 214 awake [20254.725481] LustreError: 13501:0:(tgt_handler.c:2704:tgt_brw_write()) Skipped 1 previous similar message [20267.904109] Lustre: DEBUG MARKER: == sanity test 398h: verify correctness of read [20290.929141] Lustre: DEBUG MARKER: == sanity test 398i: verify parallel dio handles ll_direct_rw_pages errors correctly ========================================================== 00:32:42 (1746160362) [20299.757421] Lustre: DEBUG MARKER: == sanity test 398j: test parallel dio where stripe size > rpc_size ========================================================== 00:32:50 (1746160370) [20322.260091] Lustre: DEBUG MARKER: == sanity test 398k: test enospc on first stripe ========= 00:33:13 (1746160393) [20337.674740] Lustre: DEBUG MARKER: SKIP: sanity test_398k 7195996 > 600000 skipping out-of-space test on OST0 [20339.269023] Lustre: DEBUG MARKER: == sanity test 398l: test enospc on intermediate stripe/RPC ========================================================== 00:33:30 (1746160410) [20348.159567] Lustre: DEBUG MARKER: SKIP: sanity test_398l 7176084 > 600000 skipping out-of-space test on OST0 [20361.701540] Lustre: DEBUG MARKER: == sanity test 398m: test RPC failures with parallel dio ========================================================== 00:33:52 (1746160432) [20362.978977] Lustre: *** cfs_fail_loc=20e, val=1*** [20362.988880] Lustre: Skipped 4 previous similar messages [20364.032741] Lustre: *** cfs_fail_loc=20e, val=1*** [20364.051552] Lustre: Skipped 3 previous similar messages [20366.096573] Lustre: *** cfs_fail_loc=20e, val=1*** [20366.103382] Lustre: Skipped 1 previous similar message [20369.209088] Lustre: *** cfs_fail_loc=20e, val=1*** [20369.217800] Lustre: Skipped 2 previous similar messages [20373.277938] Lustre: *** cfs_fail_loc=20e, val=1*** [20384.338010] Lustre: *** cfs_fail_loc=20e, val=1*** [20384.340463] Lustre: Skipped 7 previous similar messages [20408.411236] Lustre: *** cfs_fail_loc=20e, val=1*** [20408.413480] Lustre: Skipped 7 previous similar messages [20443.827487] Lustre: *** cfs_fail_loc=20f, val=1*** [20443.832296] Lustre: Skipped 35 previous similar messages [20507.928184] Lustre: *** cfs_fail_loc=20e, val=2*** [20507.938696] Lustre: Skipped 51 previous similar messages [20602.943176] Lustre: DEBUG MARKER: == sanity test 398n: test append with parallel DIO ======= 00:37:54 (1746160674) [20622.152197] Lustre: DEBUG MARKER: == sanity test 398o: right kms with DIO ================== 00:38:13 (1746160693) [20623.792138] Lustre: DEBUG MARKER: sanity test_398o: @@@@@@ FAIL: bad KMS [20631.324775] Lustre: DEBUG MARKER: == sanity test 398p: race aio with buffered i/o ========== 00:38:23 (1746160703) [20687.253529] Lustre: DEBUG MARKER: == sanity test 398q: race dio with buffered i/o ========== 00:39:18 (1746160758) [20816.316896] Lustre: DEBUG MARKER: == sanity test 398r: i/o error on file read ============== 00:41:27 (1746160887) [20817.541412] Lustre: *** cfs_fail_loc=20f, val=0*** [20817.543420] Lustre: Skipped 60 previous similar messages [20878.145635] Lustre: DEBUG MARKER: == sanity test 398s: i/o error on mirror file read ======= 00:42:29 (1746160949) [20879.696485] Lustre: DEBUG MARKER: sanity test_398s: @@@@@@ FAIL: mirror resync failed