[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 318308531 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.405653] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.408507] pid_max: default: 32768 minimum: 301 [ 0.409982] Security Framework initialized [ 0.411451] SELinux: Initializing. [ 0.412764] SELinux: Starting in permissive mode [ 0.414044] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.417596] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.420175] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.422248] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.425211] Initializing cgroup subsys memory [ 0.426779] Initializing cgroup subsys devices [ 0.427851] Initializing cgroup subsys freezer [ 0.429438] Initializing cgroup subsys net_cls [ 0.430673] Initializing cgroup subsys blkio [ 0.431737] Initializing cgroup subsys perf_event [ 0.432821] Initializing cgroup subsys hugetlb [ 0.434235] Initializing cgroup subsys pids [ 0.435310] Initializing cgroup subsys net_prio [ 0.437159] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.439845] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.441233] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.442785] tlb_flushall_shift: 6 [ 0.443731] FEATURE SPEC_CTRL Present [ 0.444383] FEATURE IBPB_SUPPORT Present [ 0.445336] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.446744] Spectre V2 : Vulnerable [ 0.447467] Speculative Store Bypass: Vulnerable [ 0.449234] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.455109] ACPI: Core revision 20130517 [ 0.457879] ACPI: All ACPI Tables successfully acquired [ 0.459622] ftrace: allocating 30294 entries in 119 pages [ 0.513716] Enabling x2apic [ 0.514553] Enabled x2apic [ 0.515700] Switched APIC routing to physical x2apic. [ 0.518796] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.520704] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.523690] TSC deadline timer enabled [ 0.523860] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.526251] ... version: 2 [ 0.527358] ... bit width: 48 [ 0.528462] ... generic registers: 4 [ 0.529494] ... value mask: 0000ffffffffffff [ 0.530948] ... max period: 00007fffffffffff [ 0.532495] ... fixed-purpose events: 3 [ 0.533546] ... event mask: 000000070000000f [ 0.535103] KVM setup paravirtual spinlock [ 0.540026] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.542787] KVM setup async PF for cpu 1 [ 0.543829] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.545987] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.548740] KVM setup async PF for cpu 2 [ 0.538343] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.549245] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.551259] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.552278] Brought up 4 CPUs [ 0.552294] KVM setup async PF for cpu 3 [ 0.552301] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.555526] smpboot: Max logical packages: 1 [ 0.556691] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.560305] devtmpfs: initialized [ 0.561465] x86/mm: Memory block size: 128MB [ 0.566357] EVM: security.selinux [ 0.567442] EVM: security.ima [ 0.568309] EVM: security.capability [ 0.571846] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.574152] NET: Registered protocol family 16 [ 0.575689] cpuidle: using governor haltpoll [ 0.577174] ACPI: bus type PCI registered [ 0.578232] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.580198] PCI: Using configuration type 1 for base access [ 0.581708] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.589641] ACPI: Added _OSI(Module Device) [ 0.590892] ACPI: Added _OSI(Processor Device) [ 0.592215] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.593558] ACPI: Added _OSI(Processor Aggregator Device) [ 0.595135] ACPI: Added _OSI(Linux-Dell-Video) [ 0.597134] ACPI: EC: Look up EC in DSDT [ 0.599840] ACPI: Interpreter enabled [ 0.600935] ACPI: (supports S0 S3 S4 S5) [ 0.602196] ACPI: Using IOAPIC for interrupt routing [ 0.603833] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.606768] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.613995] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.615950] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.618028] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.620034] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.624187] acpiphp: Slot [2] registered [ 0.625497] acpiphp: Slot [3] registered [ 0.626806] acpiphp: Slot [4] registered [ 0.628055] acpiphp: Slot [5] registered [ 0.629335] acpiphp: Slot [6] registered [ 0.630612] acpiphp: Slot [7] registered [ 0.631832] acpiphp: Slot [8] registered [ 0.633200] acpiphp: Slot [9] registered [ 0.634474] acpiphp: Slot [10] registered [ 0.635738] acpiphp: Slot [11] registered [ 0.637110] acpiphp: Slot [12] registered [ 0.638594] acpiphp: Slot [13] registered [ 0.639893] acpiphp: Slot [14] registered [ 0.641269] acpiphp: Slot [15] registered [ 0.642524] acpiphp: Slot [16] registered [ 0.643758] acpiphp: Slot [17] registered [ 0.645074] acpiphp: Slot [18] registered [ 0.646407] acpiphp: Slot [19] registered [ 0.647710] acpiphp: Slot [20] registered [ 0.648979] acpiphp: Slot [21] registered [ 0.650610] acpiphp: Slot [22] registered [ 0.651992] acpiphp: Slot [23] registered [ 0.653325] acpiphp: Slot [24] registered [ 0.654588] acpiphp: Slot [25] registered [ 0.655902] acpiphp: Slot [26] registered [ 0.657207] acpiphp: Slot [27] registered [ 0.658554] acpiphp: Slot [28] registered [ 0.659860] acpiphp: Slot [29] registered [ 0.661714] acpiphp: Slot [30] registered [ 0.663079] acpiphp: Slot [31] registered [ 0.664346] PCI host bridge to bus 0000:00 [ 0.665733] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.667856] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.669983] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.672227] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.674547] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.676970] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.678725] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.679499] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.680423] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.687959] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.692101] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.694293] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.696344] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.698543] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.700904] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.701529] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.703843] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.706504] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.708382] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.716365] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.718616] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.719554] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.725912] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.730172] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.749099] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.755497] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.762405] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.765452] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.779037] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.786405] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.792148] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.800716] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.817642] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.829039] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.834269] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.838029] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.852273] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.857403] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.860962] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.864388] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.882105] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.886716] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.893655] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.898842] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.915032] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.922789] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.925249] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.927150] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.928794] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.930605] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.933402] vgaarb: loaded [ 0.934544] SCSI subsystem initialized [ 0.936077] ACPI: bus type USB registered [ 0.937150] usbcore: registered new interface driver usbfs [ 0.938338] usbcore: registered new interface driver hub [ 0.939758] usbcore: registered new device driver usb [ 0.941329] PCI: Using ACPI for IRQ routing [ 0.942588] PCI: pci_cache_line_size set to 64 bytes [ 0.942760] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.942763] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 0.942764] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 0.943044] NetLabel: Initializing [ 0.944113] NetLabel: domain hash size = 128 [ 0.945152] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.946406] NetLabel: unlabeled traffic allowed by default [ 0.948308] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.949907] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.954835] amd_nb: Cannot enumerate AMD northbridges [ 0.957075] Switched to clocksource kvm-clock [ 0.975786] pnp: PnP ACPI init [ 0.976954] ACPI: bus type PNP registered [ 0.978504] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.978586] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.978634] pnp 00:02: [dma 2] [ 0.978674] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.978754] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.978838] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.978932] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.979335] pnp: PnP ACPI: found 6 devices [ 0.980863] ACPI: bus type PNP unregistered [ 0.992104] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.992109] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.992110] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.992112] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.992114] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 0.992177] NET: Registered protocol family 2 [ 0.993693] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 0.995749] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 0.998069] TCP: Hash tables configured (established 32768 bind 32768) [ 1.000032] TCP: reno registered [ 1.001524] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.003388] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.005515] NET: Registered protocol family 1 [ 1.007486] RPC: Registered named UNIX socket transport module. [ 1.009523] RPC: Registered udp transport module. [ 1.010513] RPC: Registered tcp transport module. [ 1.011766] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.013519] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.015138] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.016580] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.018147] PCI: CLS 0 bytes, default 64 [ 1.018288] Unpacking initramfs... [ 2.360909] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.366981] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.370411] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.376219] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.378700] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.380462] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.382278] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.385726] cryptomgr_test (50) used greatest stack depth: 14480 bytes left [ 2.386281] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.386327] Initialise system trusted keyring [ 2.426270] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.428741] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.436130] zpool: loaded [ 2.437107] zbud: loaded [ 2.438397] VFS: Disk quotas dquot_6.6.0 [ 2.439935] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.443517] NFS: Registering the id_resolver key type [ 2.445342] Key type id_resolver registered [ 2.446623] Key type id_legacy registered [ 2.447696] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.450484] Key type big_key registered [ 2.451646] SELinux: Registering netfilter hooks [ 2.454460] cryptomgr_test (56) used greatest stack depth: 13968 bytes left [ 2.457863] NET: Registered protocol family 38 [ 2.458776] Key type asymmetric registered [ 2.459756] Asymmetric key parser 'x509' registered [ 2.461251] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.463707] io scheduler noop registered [ 2.464860] io scheduler deadline registered (default) [ 2.466198] io scheduler cfq registered [ 2.467058] io scheduler mq-deadline registered [ 2.468294] io scheduler kyber registered [ 2.471430] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.473242] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.475242] intel_idle: does not run on family 6 model 62 [ 2.475372] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.477388] ACPI: Power Button [PWRF] [ 2.478915] GHES: HEST is not enabled! [ 2.533754] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.596049] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.721579] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.786133] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 2.912050] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.941178] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 2.968899] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 2.972618] Non-volatile memory driver v1.3 [ 2.974140] Linux agpgart interface v0.103 [ 2.976286] crash memory driver: version 1.1 [ 2.979131] nbd: registered device at major 43 [ 2.989149] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 2.989199] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 2.989238] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 2.989274] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 2.989313] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 2.997684] virtio_blk virtio1: [vda] 53392 512-byte logical blocks (27.3 MB/26.0 MiB) [ 3.013819] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 3.013852] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 3.013875] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 3.013940] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 3.013979] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 3.020814] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.031565] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 3.031601] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 3.031624] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 3.031646] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 3.031673] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 3.037523] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.045459] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 3.045488] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 3.045517] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 3.045540] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 3.045567] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 3.051975] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.065344] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 3.065382] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 3.065417] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 3.065461] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 3.065498] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 3.071303] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.079185] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 3.079235] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 3.079276] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 3.079325] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 3.079362] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 3.087430] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.096917] rdac: device handler registered [ 3.098969] hp_sw: device handler registered [ 3.100378] emc: device handler registered [ 3.101927] libphy: Fixed MDIO Bus: probed [ 3.106871] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.108368] ehci-pci: EHCI PCI platform driver [ 3.109766] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.111129] ohci-pci: OHCI PCI platform driver [ 3.112647] uhci_hcd: USB Universal Host Controller Interface driver [ 3.114919] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.118586] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.120157] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.122106] mousedev: PS/2 mouse device common for all mice [ 3.124812] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.128446] rtc_cmos 00:05: RTC can wake from S4 [ 3.131441] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.134416] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.138627] hidraw: raw HID events driver (C) Jiri Kosina [ 3.141096] usbcore: registered new interface driver usbhid [ 3.142612] usbhid: USB HID core driver [ 3.144043] drop_monitor: Initializing network drop monitor service [ 3.146158] Netfilter messages via NETLINK v0.30. [ 3.147834] TCP: cubic registered [ 3.148988] Initializing XFRM netlink socket [ 3.151013] NET: Registered protocol family 10 [ 3.153865] NET: Registered protocol family 17 [ 3.155445] Key type dns_resolver registered [ 3.157321] mce: Using 10 MCE banks [ 3.159174] Loading compiled-in X.509 certificates [ 3.162015] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.165225] registered taskstats version 1 [ 3.169568] modprobe (71) used greatest stack depth: 13456 bytes left [ 3.174913] Key type trusted registered [ 3.179913] Key type encrypted registered [ 3.181481] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.185508] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.189472] rtc_cmos 00:05: setting system clock to 2024-07-29 07:23:42 UTC (1722237822) [ 3.192430] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.194806] Write protecting the kernel read-only data: 12288k [ 3.197380] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.200437] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.208639] random: systemd: uninitialized urandom read (16 bytes read) [ 3.211316] random: systemd: uninitialized urandom read (16 bytes read) [ 3.213646] random: systemd: uninitialized urandom read (16 bytes read) [ 3.217113] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.220952] systemd[1]: Detected virtualization kvm. [ 3.221980] systemd[1]: Detected architecture x86-64. [ 3.223125] systemd[1]: Running in initial RAM disk. [ 3.227589] systemd[1]: No hostname configured. [ 3.228997] systemd[1]: Set hostname to . [ 3.230787] random: systemd: uninitialized urandom read (16 bytes read) [ 3.232941] systemd[1]: Initializing machine ID from random generator. [ 3.245624] ln (88) used greatest stack depth: 13008 bytes left [ 3.283097] random: systemd: uninitialized urandom read (16 bytes read) [ 3.284816] random: systemd: uninitialized urandom read (16 bytes read) [ 3.286761] random: systemd: uninitialized urandom read (16 bytes read) [ 3.288542] random: systemd: uninitialized urandom read (16 bytes read) [ 3.291778] random: systemd: uninitialized urandom read (16 bytes read) [ 3.293751] random: systemd: uninitialized urandom read (16 bytes read) [ 3.303551] systemd[1]: Reached target Timers. [ 3.306703] systemd[1]: Reached target Local File Systems. [ 3.311028] systemd[1]: Reached target Swap. [ 3.315049] systemd[1]: Created slice Root Slice. [ 3.318840] systemd[1]: Listening on udev Control Socket. [ 3.322937] systemd[1]: Created slice System Slice. [ 3.325898] systemd[1]: Reached target Slices. [ 3.329035] systemd[1]: Listening on udev Kernel Socket. [ 3.332015] systemd[1]: Listening on Journal Socket. [ 3.338889] systemd[1]: Starting Load Kernel Modules... [ 3.344267] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.350956] systemd[1]: Starting Journal Service... [ 3.354319] systemd[1]: Reached target Sockets. [ 3.359183] systemd[1]: Starting Setup Virtual Console... [ 3.364091] systemd[1]: Starting dracut cmdline hook... [ 3.369101] systemd[1]: Started Load Kernel Modules. [ 3.376810] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 3.383414] systemd[1]: Started Setup Virtual Console. [ 3.386185] tsc: Refined TSC clocksource calibration: 2399.957 MHz [ 3.389376] systemd[1]: Started Journal Service. [ 3.766211] random: fast init done [ 3.901033] libata version 3.00 loaded. [ 3.914542] ata_piix 0000:00:01.1: version 2.13 [ 3.935819] scsi host0: ata_piix [ 3.946129] scsi host1: ata_piix [ 3.947564] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.950615] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 3.971976] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 4.199987] ip (343) used greatest stack depth: 12336 bytes left [ 10.286228] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 10.689293] systemd-journald[104]: Received SIGTERM from PID 1 (systemd). [ 10.897068] SELinux: Disabled at runtime. [ 10.898579] SELinux: Unregistering netfilter hooks [ 10.959891] ip_tables: (C) 2000-2006 Netfilter Core Team [ 10.962516] systemd[1]: Inserted module 'ip_tables' [ 11.360179] systemd-journald[566]: Received request to flush runtime journal from PID 1 [ 11.454113] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 11.464831] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 11.514138] cryptd: max_cpu_qlen set to 1000 [ 11.546599] AVX version of gcm_enc/dec engaged. [ 11.547958] AES CTR mode by8 optimization enabled [ 11.581904] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 11.595448] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 11.601970] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 11.607135] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 11.707533] EDAC MC: Ver: 3.0.0 [ 11.715364] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 11.715374] EDAC sbridge: Ver: 1.1.2 [ 13.427209] mount.nfs (769) used greatest stack depth: 10704 bytes left [ 20.902279] device-mapper: uevent: version 1.0.3 [ 20.903478] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 25.219198] libcfs: loading out-of-tree module taints kernel. [ 25.220996] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 25.236242] LNet: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 25.240595] alg: No test for adler32 (adler32-zlib) [ 26.011978] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 30.849199] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 31.144292] Lustre: Lustre: Build Version: 2.15.5_1_g87cf75c [ 31.299787] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 31.301588] LNet: Accept secure, port 988 [ 32.843166] Key type lgssc registered [ 33.134527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 36.178024] icp: module license 'CDDL' taints kernel. [ 36.179315] Disabling lock debugging due to kernel taint [ 38.905730] ZFS: Loaded module v0.8.6-1, ZFS pool version 5000, ZFS filesystem version 5 [ 41.684643] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 46.021507] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 48.457529] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 48.460483] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 50.802820] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 50.805872] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 53.582872] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 56.275459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 56.298435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 57.361748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 57.369024] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 57.398326] Lustre: lustre-MDT0000: new disk, initializing [ 57.415101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 57.420822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 57.440928] mount.lustre (6808) used greatest stack depth: 10064 bytes left [ 58.207910] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 62.399461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 62.422802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 62.446553] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 62.454650] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 62.457573] Lustre: Skipped 1 previous similar message [ 62.492687] Lustre: lustre-MDT0001: new disk, initializing [ 62.505759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 62.512568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 62.515317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 63.365876] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 64.467597] random: crng init done [ 67.469005] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 67.472368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 67.494014] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 67.496978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 67.499399] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 67.560239] Lustre: lustre-OST0000: new disk, initializing [ 67.561785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 67.572520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 68.428365] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 72.807838] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 72.811533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 72.831752] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 72.835935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 72.839231] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 72.867861] Lustre: lustre-OST0001: new disk, initializing [ 72.870127] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 72.882353] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 73.723393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 73.726443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 73.809237] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 78.622902] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 85.778781] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 91.575004] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing check_logdir /tmp/testlogs/ [ 92.528183] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing yml_node [ 93.413097] Lustre: DEBUG MARKER: Client: 2.15.5.1 [ 94.011044] Lustre: DEBUG MARKER: MDS: 2.15.5.1 [ 94.609667] Lustre: DEBUG MARKER: OSS: 2.15.5.1 [ 94.975908] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Mon Jul 29 03:25:14 EDT 2024 [ 96.426314] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42b 42c 407 118c 118d [ 96.782534] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 115 135 136 230d 300o [ 98.143515] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing check_config_client /mnt/lustre [ 103.169042] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 104.032758] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 104.738450] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 106.487703] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 03:25:25 (1722237925) [ 107.350948] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 107.772941] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 03:25:26 (1722237926) [ 109.637630] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 03:25:28 (1722237928) [ 126.984589] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 03:25:46 (1722237946) [ 128.353090] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 03:25:47 (1722237947) [ 128.623083] Lustre: *** cfs_fail_loc=15b, val=0*** [ 128.624989] Lustre: *** cfs_fail_loc=15b, val=0*** [ 129.917196] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 03:25:49 (1722237949) [ 131.448159] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 03:25:50 (1722237950) [ 131.719577] Lustre: *** cfs_fail_loc=19a, val=0*** [ 132.172681] LustreError: 6859:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 132.176374] LustreError: 6859:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 132.413033] Lustre: *** cfs_fail_loc=19a, val=0*** [ 132.414255] Lustre: Skipped 2 previous similar messages [ 133.128302] LustreError: 6859:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 133.132086] LustreError: 6859:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message [ 133.135597] LustreError: 6859:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 133.140254] LustreError: 6859:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1 previous similar message [ 133.584403] Lustre: *** cfs_fail_loc=19a, val=0*** [ 133.585928] Lustre: Skipped 4 previous similar messages [ 134.529699] LustreError: 6859:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 134.534120] LustreError: 6859:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message [ 134.537235] LustreError: 6859:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 134.542352] LustreError: 6859:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1 previous similar message [ 135.632841] Lustre: *** cfs_fail_loc=19a, val=0*** [ 135.633988] Lustre: Skipped 8 previous similar messages [ 137.437170] LustreError: 6859:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 137.441279] LustreError: 6859:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 4 previous similar messages [ 137.444115] LustreError: 6859:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 137.448175] LustreError: 6859:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 4 previous similar messages [ 139.706363] Lustre: *** cfs_fail_loc=19a, val=0*** [ 139.707745] Lustre: Skipped 17 previous similar messages [ 142.414389] LustreError: 6859:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 142.416908] LustreError: 6859:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 7 previous similar messages [ 142.419998] LustreError: 6859:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 142.423691] LustreError: 6859:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 7 previous similar messages [ 147.835180] Lustre: *** cfs_fail_loc=19a, val=0*** [ 147.836909] Lustre: Skipped 35 previous similar messages [ 151.234594] LustreError: 8063:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0001-osd: fail to cancel 1 llog-records: rc = -5 [ 151.237754] LustreError: 8063:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 13 previous similar messages [ 151.240396] LustreError: 8063:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0001-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 151.243801] LustreError: 8063:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 13 previous similar messages [ 157.306633] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 03:26:16 (1722237976) [ 157.559298] Lustre: *** cfs_fail_loc=188, val=0*** [ 158.125814] Lustre: *** cfs_fail_loc=189, val=0*** [ 159.795115] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 160.183551] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 03:26:19 (1722237979) [ 161.686857] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 03:26:20 (1722237980) [ 163.122691] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 03:26:22 (1722237982) [ 225.515883] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 03:27:24 (1722238044) [ 230.153613] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 03:27:29 (1722238049) [ 232.166476] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 232.586385] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 03:27:31 (1722238051) [ 234.646055] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 03:27:33 (1722238053) [ 256.924383] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 03:27:56 (1722238076) [ 257.581804] Lustre: *** cfs_fail_loc=725, val=0*** [ 258.343240] Lustre: *** cfs_fail_loc=725, val=0*** [ 260.150668] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 03:27:59 (1722238079) [ 262.589090] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 03:28:01 (1722238081) [ 300.501002] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 03:28:39 (1722238119) [ 310.950964] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 03:28:50 (1722238130) [ 313.254783] Lustre: *** cfs_fail_loc=513, val=0*** [ 313.256493] LustreError: 15974:0:(service.c:2115:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1805897274334272 [ 313.756758] Lustre: Failing over lustre-OST0000 [ 313.760669] LustreError: 31110:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [ 313.770646] Lustre: server umount lustre-OST0000 complete [ 314.122543] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 314.122640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.122901] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 314.132055] LustreError: Skipped 1 previous similar message [ 315.925210] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 315.930806] LustreError: Skipped 1 previous similar message [ 319.130631] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.135406] LustreError: Skipped 1 previous similar message [ 324.138671] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.142758] LustreError: Skipped 2 previous similar messages [ 326.015402] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 326.019045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 326.060406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 326.930621] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 327.590148] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 328.484673] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 328.484689] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.115@tcp (at 0@lo) [ 328.486591] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2519 to 0x0:2561 [ 329.840768] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 330.195547] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 333.283417] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 03:29:12 (1722238152) [ 334.758945] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 03:29:13 (1722238153) [ 336.217059] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 03:29:15 (1722238155) [ 337.728002] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 03:29:16 (1722238156) [ 339.312591] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 03:29:18 (1722238158) [ 340.969736] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 03:29:20 (1722238160) [ 342.647675] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 03:29:21 (1722238161) [ 344.324246] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 03:29:23 (1722238163) [ 346.080593] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 03:29:25 (1722238165) [ 348.132317] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 03:29:27 (1722238167) [ 350.846542] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 03:29:30 (1722238170) [ 351.402847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.408839] Lustre: Skipped 1 previous similar message [ 351.411166] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 351.414035] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.204.115@tcp (at 0@lo) [ 351.414251] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 351.419785] Lustre: Skipped 1 previous similar message [ 351.669949] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:33 [ 351.918890] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 351.921472] Lustre: Skipped 1 previous similar message [ 351.923342] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2564 to 0x0:2593 [ 352.184102] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2515 to 0x0:2561 [ 355.679356] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 359.097587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.101870] Lustre: Skipped 3 previous similar messages [ 359.103447] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 359.105931] Lustre: Skipped 1 previous similar message [ 359.107524] LustreError: 167-0: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 359.111286] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.204.115@tcp (at 0@lo) [ 359.111513] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97 [ 359.116083] Lustre: Skipped 3 previous similar messages [ 360.036248] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 360.086521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 360.971156] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 361.021333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 364.390485] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 367.880848] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.886464] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 367.890215] LustreError: 167-0: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 367.896171] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.204.115@tcp (at 0@lo) [ 367.896418] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1034 to 0x2c0000400:1057 [ 368.894863] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 368.954565] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 369.944501] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 369.999924] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 373.833994] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 377.229389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.233514] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 377.236020] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 377.241428] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.115@tcp (at 0@lo) [ 377.241773] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4594 to 0x0:4609 [ 378.167286] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 378.216749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 379.143158] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 379.194943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 382.516461] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 385.580359] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 385.584407] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 385.586784] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 385.589965] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.204.115@tcp (at 0@lo) [ 385.590126] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2566 to 0x0:2593 [ 386.462394] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 386.519382] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 387.357375] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 387.398483] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 388.807742] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 03:30:08 (1722238208) [ 390.325053] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 03:30:09 (1722238209) [ 391.732794] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 03:30:10 (1722238210) [ 396.659424] LustreError: 10775:0:(qmt_pool.c:1402:qmt_pool_add_rem()) add to: can't lustre-QMT0000 lustre-OST0000_UUID pool test_65n: rc = -17 [ 401.103089] LustreError: 11314:0:(qmt_pool.c:1402:qmt_pool_add_rem()) remove: can't lustre-QMT0000 lustre-OST0000_UUID pool test_65n: rc = -22 [ 401.106210] LustreError: 11314:0:(qmt_pool.c:1402:qmt_pool_add_rem()) Skipped 1 previous similar message [ 402.345317] LustreError: 11391:0:(qmt_pool.c:1402:qmt_pool_add_rem()) remove: can't lustre-QMT0000 lustre-OST0001_UUID pool test_65n: rc = -22 [ 404.745133] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 03:30:23 (1722238223) [ 407.476620] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 03:30:26 (1722238226) [ 407.941700] Lustre: *** cfs_fail_loc=217, val=0*** [ 408.767619] Lustre: *** cfs_fail_loc=217, val=0*** [ 408.768674] Lustre: Skipped 1 previous similar message [ 410.454261] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 410.811567] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 03:30:30 (1722238230) [ 412.263268] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 03:30:31 (1722238231) [ 413.849846] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 03:30:33 (1722238233) [ 441.341429] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 03:31:00 (1722238260) [ 442.697402] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 03:31:01 (1722238261) [ 444.027556] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 03:31:03 (1722238263) [ 445.426758] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 03:31:04 (1722238264) [ 454.626201] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 03:31:13 (1722238273) [ 461.461016] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 03:31:20 (1722238280) [ 463.142500] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 03:31:22 (1722238282) [ 463.294419] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.15@tcp inode [0x200000407:0xc16:0x0] object 0x0:2857 extent [0-4194303]: client csum 5e539644, server csum 5e539643 [ 464.749067] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 465.862597] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.204.15@tcp inode [0x200000407:0xc16:0x0] object 0x0:2857 extent [0-4194303], client returned csum e53aad86 (type 1), server csum 7b28f926 (type 1) [ 466.458443] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 467.565924] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.204.15@tcp inode [0x200000407:0xc16:0x0] object 0x0:2857 extent [0-4194303], client returned csum b194e6ab (type 2), server csum d645e80b (type 2) [ 468.158987] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 469.257619] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.204.15@tcp inode [0x200000407:0xc16:0x0] object 0x0:2857 extent [0-4194303], client returned csum 213f4650 (type 4), server csum 5e539643 (type 4) [ 469.866366] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 471.592545] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 472.686616] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.204.15@tcp inode [0x200000407:0xc16:0x0] object 0x0:2857 extent [0-4194303], client returned csum c81008ad (type 20), server csum bfcb084c (type 20) [ 472.692315] LustreError: Skipped 1 previous similar message [ 473.294369] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 475.006876] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 476.612031] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 478.020843] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 03:31:37 (1722238297) [ 479.904443] Lustre: 24057:0:(tgt_handler.c:1903:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000407:0xc17:0x0]:[0-1048575]-d8303b1b-d7cc6ddf [ 479.909850] LustreError: dumping log to /tmp/lustre-log.1722238299.24057 [ 479.925752] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.204.15@tcp inode [0x200000407:0xc17:0x0] object 0x0:4874 extent [0-1048575], client returned csum d8303b1b (type 4), server csum d7cc6ddf (type 4) [ 479.931103] LustreError: Skipped 2 previous similar messages [ 487.571864] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 03:31:46 (1722238306) [ 487.681639] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.204.15@tcp inode [0x200000407:0xc19:0x0] object 0x0:4875 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 489.901086] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.204.15@tcp inode [0x200000407:0xc19:0x0] object 0x0:4875 extent [0-4194303], client returned csum f5a99216 (type 4), server csum de2bf73e (type 4) [ 491.352570] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 03:31:50 (1722238310) [ 491.713638] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 491.808554] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.15@tcp inode [0x200000407:0xc1a:0x0] object 0x0:2858 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 491.815493] LustreError: Skipped 1 previous similar message [ 494.890528] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.15@tcp inode [0x200000407:0xc1a:0x0] object 0x0:2858 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 494.896152] LustreError: Skipped 3 previous similar messages [ 501.985401] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.15@tcp inode [0x200000407:0xc1a:0x0] object 0x0:2858 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 501.990196] LustreError: Skipped 3 previous similar messages [ 513.088157] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.15@tcp inode [0x200000407:0xc1a:0x0] object 0x0:2858 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 513.095211] LustreError: Skipped 3 previous similar messages [ 531.223995] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.15@tcp inode [0x200000407:0xc1a:0x0] object 0x0:2858 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 531.228050] LustreError: Skipped 5 previous similar messages [ 537.689297] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 565.088788] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.15@tcp inode [0x200000407:0xc1a:0x0] object 0x0:2858 extent [4194304-8388607]: client csum 7d33b9a0, server csum 7d33b99f [ 565.094468] LustreError: Skipped 16 previous similar messages [ 583.797311] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 631.430665] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.15@tcp inode [0x200000407:0xc1a:0x0] object 0x0:2858 extent [4194304-8388607]: client csum de2bf73f, server csum de2bf73e [ 631.437382] LustreError: Skipped 27 previous similar messages [ 631.893091] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 677.698955] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 723.039931] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 761.682580] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.15@tcp inode [0x200000407:0xc1a:0x0] object 0x0:2858 extent [4194304-8388607]: client csum a30822f0, server csum a30822ef [ 761.686440] LustreError: Skipped 63 previous similar messages [ 768.229769] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 814.209575] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 815.836328] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 03:37:15 (1722238635) [ 816.300514] Lustre: *** cfs_fail_loc=21a, val=0*** [ 817.147411] Lustre: *** cfs_fail_loc=21b, val=0*** [ 820.272696] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 03:37:19 (1722238639) [ 820.547973] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 821.135112] Lustre: Modifying parameter lustre.osc.lustre*.checksums in log params [ 822.781564] Lustre: Disabling parameter lustre.osc.lustre*.checksums in log params [ 825.465732] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 827.072996] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 03:37:26 (1722238646) [ 827.597779] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 828.111508] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 829.831334] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid 40 [ 844.485835] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid in IDLE state after 14 sec [ 845.946150] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid 40 [ 846.361240] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid in FULL state after 0 sec [ 846.797659] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 848.392457] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid 40 [ 870.070458] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid in IDLE state after 21 sec [ 871.522807] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid 40 [ 871.882020] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid in FULL state after 0 sec [ 872.254279] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 873.625425] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid 40 [ 894.489207] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid in IDLE state after 20 sec [ 896.145348] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid 40 [ 896.580173] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid in FULL state after 0 sec [ 897.111658] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 899.295915] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid 40 [ 920.186223] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid in IDLE state after 20 sec [ 921.998366] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid 40 [ 922.488815] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid in FULL state after 0 sec [ 922.991522] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 924.741735] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid 40 [ 944.584371] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid in IDLE state after 19 sec [ 946.360241] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid 40 [ 946.863880] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid in FULL state after 0 sec [ 947.354582] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 949.099472] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid 40 [ 969.982423] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid in IDLE state after 20 sec [ 971.696914] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid 40 [ 972.144926] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid in FULL state after 0 sec [ 972.619885] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 974.301874] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid 40 [ 995.161024] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid in IDLE state after 20 sec [ 996.822850] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid 40 [ 997.257608] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b590e000.ost_server_uuid in FULL state after 0 sec [ 999.103353] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 999.609359] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 03:40:18 (1722238818) [ 1001.464503] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 03:40:20 (1722238820) [ 1002.050059] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1002.527440] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1003.030727] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1003.473147] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1005.485206] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1006.001582] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 03:40:25 (1722238825) [ 1009.614086] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 03:40:28 (1722238828) [ 1012.068442] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ================================================================================= 03:40:31 (1722238831) [ 1026.495669] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 03:40:45 (1722238845) [ 1029.878176] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 03:40:48 (1722238848) [ 1030.308635] Lustre: *** cfs_fail_loc=228, val=0*** [ 1032.401640] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 03:40:51 (1722238851) [ 1032.815569] Lustre: *** cfs_fail_loc=228, val=0*** [ 1034.960523] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 03:40:54 (1722238854) [ 1042.019009] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ===================================================================== 03:41:01 (1722238861) [ 1044.852187] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 03:41:03 (1722238863) [ 1067.440556] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 03:41:26 (1722238886) [ 1071.493580] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 03:41:30 (1722238890) [ 1080.986119] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 03:41:40 (1722238900) [ 1122.294301] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 03:42:21 (1722238941) [ 1130.922615] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 03:42:30 (1722238950) [ 1133.422834] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 03:42:32 (1722238952) [ 1141.030530] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 03:42:40 (1722238960) [ 1143.349151] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 03:42:42 (1722238962) [ 1145.223299] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 03:42:44 (1722238964) [ 1153.026058] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 03:42:52 (1722238972) [ 1155.108215] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 03:42:54 (1722238974) [ 1158.347587] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 03:42:57 (1722238977) [ 1160.386860] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 03:42:59 (1722238979) [ 1165.241907] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 03:43:04 (1722238984) [ 1170.483500] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 03:43:09 (1722238989) [ 1171.011449] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 1171.529374] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 1172.041712] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 1172.536048] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 1174.454650] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 03:43:13 (1722238993) [ 1175.269558] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 1175.794815] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 1176.287358] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 1176.822476] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 1177.383461] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 1179.488211] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 03:43:18 (1722238998) [ 1181.226240] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 03:43:20 (1722239000) [ 1185.108450] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 03:43:24 (1722239004) [ 1187.073284] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 03:43:26 (1722239006) [ 1188.897467] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 03:43:28 (1722239008) [ 1190.730371] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 03:43:29 (1722239009) [ 1192.757028] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 03:43:31 (1722239011) [ 1194.664383] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 03:43:33 (1722239013) [ 1196.612730] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 03:43:35 (1722239015) [ 1198.961390] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 03:43:38 (1722239018) [ 1201.080936] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 03:43:40 (1722239020) [ 1202.885088] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 03:43:42 (1722239022) [ 1236.026669] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 03:44:15 (1722239055) [ 1251.478280] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 03:44:30 (1722239070) [ 1253.319476] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 03:44:32 (1722239072) [ 1482.333056] Lustre: 6831:0:(osd_handler.c:1950:osd_trans_start()) lustre-MDT0000: credits 64189 > trans_max 3200 [ 1482.335909] Lustre: 6831:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 1000/4000/0, destroy: 1/4/0 [ 1482.338288] Lustre: 6831:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 1004/148/0 [ 1482.341046] Lustre: 6831:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 5001/43010/0, punch: 0/0/0, quota 0/0/0 [ 1482.344840] Lustre: 6831:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 1001/17016/0, delete: 2/5/0 [ 1482.347480] Lustre: 6831:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 1482.350180] Pid: 6831, comm: mdt00_002 3.10.0-7.9-debug #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 1482.352686] Call Trace: [ 1482.353748] [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [ 1482.355345] [<0>] libcfs_debug_dumpstack+0x26/0x30 [libcfs] [ 1482.357198] [<0>] osd_trans_start+0x5ba/0x5e0 [osd_ldiskfs] [ 1482.359182] [<0>] top_trans_start+0x763/0xa10 [ptlrpc] [ 1482.360815] [<0>] lod_trans_start+0x34/0x40 [lod] [ 1482.362086] [<0>] mdd_trans_start+0x14/0x20 [mdd] [ 1482.363669] [<0>] mdd_unlink+0x5e3/0xdb0 [mdd] [ 1482.364900] [<0>] mdt_reint_unlink+0xe32/0x1df0 [mdt] [ 1482.366596] [<0>] mdt_reint_rec+0x87/0x240 [mdt] [ 1482.367940] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [ 1482.369772] [<0>] mdt_reint+0x67/0x150 [mdt] [ 1482.371442] [<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc] [ 1482.373408] [<0>] ptlrpc_server_handle_request+0x250/0xc30 [ptlrpc] [ 1482.375527] [<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc] [ 1482.377156] [<0>] kthread+0xe4/0xf0 [ 1482.378229] [<0>] ret_from_fork_nospec_begin+0x7/0x21 [ 1482.379629] [<0>] 0xfffffffffffffffe [ 1482.380734] Lustre: 6831:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 3200 < left 43010, rollback = 7 [ 1484.123394] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 03:48:23 (1722239303) [ 1485.101860] Lustre: lustre-MDD0000: changelog on [ 1485.897266] Lustre: lustre-MDD0001: changelog on [ 1487.601391] Lustre: lustre-MDD0001: changelog off [ 1488.326713] Lustre: lustre-MDD0000: changelog off [ 1489.012391] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 03:48:28 (1722239308) [ 1489.114121] Lustre: lustre-OST0000: Client 0ef13c0f-64db-436c-a547-e4268e117767 (at 192.168.204.15@tcp) reconnecting [ 1490.505918] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800afc93000.ost_server_uuid 40 [ 1490.870100] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800afc93000.ost_server_uuid in FULL state after 0 sec [ 1492.563930] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 03:48:31 (1722239311) [ 1494.213843] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 03:48:33 (1722239313) [ 1494.605860] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [ 1495.073051] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 03:48:34 (1722239314) [ 1496.836121] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 03:48:36 (1722239316) [ 1498.530225] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 03:48:37 (1722239317) [ 1500.154060] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 03:48:39 (1722239319) [ 1511.754760] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 03:48:50 (1722239330) [ 1513.312324] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 03:48:52 (1722239332) [ 1515.011859] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 03:48:54 (1722239334) [ 1517.934966] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 03:48:57 (1722239337) [ 1519.864359] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [ 1520.311967] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 03:48:59 (1722239339) [ 1576.606866] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 03:49:55 (1722239395) [ 1577.373914] Lustre: *** cfs_fail_loc=147, val=0*** [ 1579.486059] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 03:49:58 (1722239398) [ 1581.007954] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 03:50:00 (1722239400) [ 1582.591020] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 03:50:01 (1722239401) [ 1583.003118] Lustre: *** cfs_fail_loc=217, val=0*** [ 1583.004446] Lustre: Skipped 29 previous similar messages [ 1584.774736] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 1585.157314] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 1585.544630] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 03:50:04 (1722239404) [ 1587.119545] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 03:50:06 (1722239406) [ 1588.726295] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 03:50:07 (1722239407) [ 1589.123912] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1590.014130] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1592.015983] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1595.022086] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1599.027124] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1600.908790] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 03:50:20 (1722239420) [ 1608.686176] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 03:50:27 (1722239427) [ 1609.089489] Lustre: *** cfs_fail_loc=220, val=0*** [ 1609.090724] Lustre: Skipped 3 previous similar messages [ 1610.901374] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 03:50:30 (1722239430) [ 1620.421464] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 03:50:39 (1722239439) [ 1621.925143] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 03:50:41 (1722239441) [ 1623.396645] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 03:50:42 (1722239442) [ 1627.842000] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 03:50:46 (1722239446) [ 1629.605723] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 03:50:48 (1722239448) [ 1631.384051] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 03:50:50 (1722239450) [ 1633.133326] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ========================================================== 03:50:52 (1722239452) [ 1634.634235] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [ 1638.453013] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 03:50:57 (1722239457) [ 1640.883695] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 03:51:00 (1722239460) [ 1643.230498] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 03:51:02 (1722239462) [ 1645.630369] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 03:51:04 (1722239464) [ 1647.907021] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 03:51:07 (1722239467) [ 1657.247280] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 03:51:16 (1722239476) [ 1666.697021] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 03:51:25 (1722239485) [ 1715.773083] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 03:52:14 (1722239534) [ 1717.384587] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 03:52:16 (1722239536) [ 1718.307968] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 0 sec [ 1718.840947] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 1724.077326] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 2 sec [ 1724.968546] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 1 sec [ 1767.533610] Lustre: DEBUG MARKER: ls -l 10000 files without statahead: 23 sec [ 1772.038282] Lustre: DEBUG MARKER: ls -l 10000 files with statahead: 4 sec [ 1772.412403] Lustre: DEBUG MARKER: ls -l done [ 1786.831366] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity/: 15 seconds [ 1787.205508] Lustre: DEBUG MARKER: rm done [ 1789.049748] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 03:53:28 (1722239608) [ 1789.435231] Lustre: DEBUG MARKER: SKIP: sanity test_123ab Test must be statx() syscall supported [ 1789.893171] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 03:53:29 (1722239609) [ 1790.281318] Lustre: DEBUG MARKER: SKIP: sanity test_123ac Test must be statx() syscall supported [ 1790.744789] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 03:53:29 (1722239609) [ 1796.428675] Lustre: DEBUG MARKER: ls done [ 1799.295805] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 03:53:38 (1722239618) [ 1801.489228] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 03:53:40 (1722239620) [ 1801.922439] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 1807.690019] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800acb0b000 [ 1808.071746] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800acb0b000 [ 1808.461139] Lustre: DEBUG MARKER: LRU=1004 [ 1808.879716] Lustre: DEBUG MARKER: LIMIT=46624 [ 1809.250540] Lustre: DEBUG MARKER: LVF=5572500 [ 1809.629730] Lustre: DEBUG MARKER: OLD_LVF=100 [ 1810.004277] Lustre: DEBUG MARKER: Sleep 50 sec [ 1860.457875] Lustre: DEBUG MARKER: Dropped 390 locks in 50s [ 1860.818390] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 1865.412999] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 03:54:44 (1722239684) [ 1881.775569] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 1903.891869] Lustre: DEBUG MARKER: ls -la time: 22 seconds [ 1904.267773] Lustre: DEBUG MARKER: lru_size = 400 [ 1935.321997] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 1941.482879] Lustre: DEBUG MARKER: ls -la time: 5 seconds [ 1941.858488] Lustre: DEBUG MARKER: lru_size = 4052 [ 1942.222841] Lustre: DEBUG MARKER: ls -la is 77% faster with lru resize enabled [ 1952.514109] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 03:56:11 (1722239771) [ 1974.623127] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize diasbaled ========================================================== 03:56:33 (1722239793) [ 1996.730548] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 03:56:55 (1722239815) [ 1998.379509] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 03:56:57 (1722239817) [ 1999.980847] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 03:56:59 (1722239819) [ 2001.754741] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 03:57:00 (1722239820) [ 2003.388359] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [ 2007.323324] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 03:57:06 (1722239826) [ 2008.947104] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 03:57:08 (1722239828) [ 2012.455836] Lustre: 6829:0:(osd_handler.c:586:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 1805, FID: [0x20000040b:0x2408:0x0]) is approaching max size limit [ 2014.314295] Lustre: 8042:0:(osd_handler.c:582:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 1805, FID: [0x20000040b:0x2408:0x0]) has reached max size limit [ 2021.716669] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 03:57:20 (1722239840) [ 2023.308968] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 03:57:22 (1722239842) [ 2024.955170] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 03:57:24 (1722239844) [ 2026.682289] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 03:57:25 (1722239845) [ 2027.037455] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [ 2027.443642] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 03:57:26 (1722239846) [ 2034.609465] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 03:57:33 (1722239853) [ 2036.176409] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 03:57:35 (1722239855) [ 2039.277516] Lustre: 23302:0:(osd_handler.c:1950:osd_trans_start()) lustre-MDT0000: credits 12989 > trans_max 3200 [ 2039.280454] Lustre: 23302:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 200/800/0, destroy: 1/4/0 [ 2039.282273] Lustre: 23302:0:(osd_handler.c:1879:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2039.284447] Lustre: 23302:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 204/148/0 [ 2039.286864] Lustre: 23302:0:(osd_handler.c:1886:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2039.288952] Lustre: 23302:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 1001/8610/0, punch: 0/0/0, quota 0/0/0 [ 2039.291441] Lustre: 23302:0:(osd_handler.c:1896:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2039.293380] Lustre: 23302:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 201/3416/0, delete: 2/5/0 [ 2039.295933] Lustre: 23302:0:(osd_handler.c:1903:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2039.298050] Lustre: 23302:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 2039.300041] Lustre: 23302:0:(osd_handler.c:1910:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2039.302390] Pid: 23302, comm: mdt00_006 3.10.0-7.9-debug #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 2039.304124] Call Trace: [ 2039.304663] [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [ 2039.305633] [<0>] libcfs_debug_dumpstack+0x26/0x30 [libcfs] [ 2039.306810] [<0>] osd_trans_start+0x5ba/0x5e0 [osd_ldiskfs] [ 2039.307953] [<0>] top_trans_start+0x763/0xa10 [ptlrpc] [ 2039.309016] [<0>] lod_trans_start+0x34/0x40 [lod] [ 2039.310001] [<0>] mdd_trans_start+0x14/0x20 [mdd] [ 2039.311268] [<0>] mdd_unlink+0x5e3/0xdb0 [mdd] [ 2039.312289] [<0>] mdt_reint_unlink+0xe32/0x1df0 [mdt] [ 2039.313360] [<0>] mdt_reint_rec+0x87/0x240 [mdt] [ 2039.314577] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [ 2039.315656] [<0>] mdt_reint+0x67/0x150 [mdt] [ 2039.316721] [<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc] [ 2039.318272] [<0>] ptlrpc_server_handle_request+0x250/0xc30 [ptlrpc] [ 2039.320154] [<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc] [ 2039.321236] [<0>] kthread+0xe4/0xf0 [ 2039.322374] [<0>] ret_from_fork_nospec_begin+0x7/0x21 [ 2039.323793] [<0>] 0xfffffffffffffffe [ 2039.324709] Lustre: 23302:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 3203 < left 8610, rollback = 7 [ 2039.327022] Lustre: 23302:0:(osd_internal.h:1333:osd_trans_exec_op()) Skipped 4000 previous similar messages [ 2039.823838] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 03:57:39 (1722239859) [ 2041.413396] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 03:57:40 (1722239860) [ 2043.020894] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 03:57:42 (1722239862) [ 2044.563780] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 03:57:43 (1722239863) [ 2046.279223] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 03:57:45 (1722239865) [ 2047.947997] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 03:57:47 (1722239867) [ 2052.671902] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 03:57:51 (1722239871) [ 2055.902296] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 03:57:55 (1722239875) [ 2074.335761] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 03:58:13 (1722239893) [ 2080.827286] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 03:58:20 (1722239900) [ 2083.544771] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 03:58:22 (1722239902) [ 2089.898636] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.898869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.898871] Lustre: Skipped 1 previous similar message [ 2089.904590] Lustre: Skipped 2 previous similar messages [ 2092.542088] LustreError: 8182:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [ 2092.544342] LustreError: 8182:0:(lprocfs_jobstats.c:137:job_stat_exit()) Skipped 7 previous similar messages [ 2092.592775] Lustre: server umount lustre-MDT0000 complete [ 2094.706804] LustreError: 8051:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1722239914 with bad export cookie 6408237699696776805 [ 2094.708209] LustreError: 166-1: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2094.717250] LustreError: 8051:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2094.743525] LustreError: 8784:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [ 2094.745911] LustreError: 8784:0:(lprocfs_jobstats.c:137:job_stat_exit()) Skipped 48 previous similar messages [ 2094.817847] Lustre: server umount lustre-MDT0001 complete [ 2101.906216] Lustre: 3322:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1722239914/real 1722239914] req@ffff8800772a04c0 x1805897283607296/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1722239921 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 2101.911735] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.914269] Lustre: Skipped 1 previous similar message [ 2103.483212] Lustre: 9393:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1722239916/real 1722239916] req@ffff88009afaed40 x1805897283607552/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1722239922 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2103.492686] LustreError: 9393:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [ 2103.494540] LustreError: 9393:0:(lprocfs_jobstats.c:137:job_stat_exit()) Skipped 26 previous similar messages [ 2103.510370] Lustre: server umount lustre-OST0000 complete [ 2105.639179] LustreError: 9991:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [ 2105.642367] LustreError: 9991:0:(lprocfs_jobstats.c:137:job_stat_exit()) Skipped 14 previous similar messages [ 2105.733334] Lustre: server umount lustre-OST0001 complete [ 2107.356361] device-mapper: core: cleaned up [ 2109.932034] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2110.445459] Key type lgssc unregistered [ 2110.511354] LNet: 10919:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.513342] LNet: Removed LNI 192.168.204.115@tcp [ 2115.366317] LNet: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2115.370905] alg: No test for adler32 (adler32-zlib) [ 2116.141799] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2116.346900] Lustre: Lustre: Build Version: 2.15.5_1_g87cf75c [ 2116.385540] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2116.387377] LNet: Accept secure, port 988 [ 2117.912214] Key type lgssc registered [ 2118.246793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2121.728126] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2122.895519] device-mapper: uevent: version 1.0.3 [ 2122.897538] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 2125.303302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.449108] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.467009] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.327240] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2130.165601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.252951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2131.161727] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2134.283853] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2134.287240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2134.377729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2134.390512] mount.lustre (15559) used greatest stack depth: 9248 bytes left [ 2135.238384] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2135.376720] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2135.381434] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:10048 to 0x280000400:10081 [ 2138.075244] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2138.078335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2138.111126] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2138.864432] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:11857 to 0x2c0000400:11873 [ 2138.979068] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2140.865207] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17750 to 0x0:17793 [ 2140.865346] Lustre: lustre-OST0001: deleting orphan objects from 0x0:15604 to 0x0:15649 [ 2146.583386] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2147.483159] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2154.198407] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 03:59:33 (1722239973) [ 2170.298925] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2170.303827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.311457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2170.906534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2170.906555] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.911754] Lustre: Skipped 3 previous similar messages [ 2173.467822] LustreError: 20537:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [ 2173.496658] Lustre: server umount lustre-MDT0000 complete [ 2175.613586] LustreError: 20538:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1722239994 with bad export cookie 8618260474338304927 [ 2175.614306] LustreError: 166-1: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2175.619396] LustreError: 20538:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2175.640230] LustreError: 21137:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [ 2175.642491] LustreError: 21137:0:(lprocfs_jobstats.c:137:job_stat_exit()) Skipped 2 previous similar messages [ 2175.704414] Lustre: server umount lustre-MDT0001 complete [ 2182.914172] Lustre: 11440:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1722239995/real 1722239995] req@ffff880137314c00 x1805899462146048/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1722240002 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 2182.920654] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.923412] Lustre: Skipped 2 previous similar messages [ 2183.812292] Lustre: 21728:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1722239997/real 1722239997] req@ffff88007a543440 x1805899462146176/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1722240003 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2183.820122] LustreError: 21728:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [ 2183.821945] LustreError: 21728:0:(lprocfs_jobstats.c:137:job_stat_exit()) Skipped 1 previous similar message [ 2183.830922] Lustre: server umount lustre-OST0000 complete [ 2185.967372] LustreError: 22326:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [ 2186.028633] Lustre: server umount lustre-OST0001 complete [ 2187.771597] device-mapper: core: cleaned up [ 2190.566348] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2191.234569] Key type lgssc unregistered [ 2191.292616] LNet: 23252:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.294901] LNet: Removed LNI 192.168.204.115@tcp [ 2195.665166] LNet: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2195.669667] alg: No test for adler32 (adler32-zlib) [ 2196.444449] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2196.643634] Lustre: Lustre: Build Version: 2.15.5_1_g87cf75c [ 2196.686715] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2196.689054] LNet: Accept secure, port 988 [ 2198.219188] Key type lgssc registered [ 2198.404662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2201.538440] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2202.537394] device-mapper: uevent: version 1.0.3 [ 2202.538666] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 2204.344982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.464311] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.478389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.304481] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2209.043014] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.126109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2209.980411] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2212.832087] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2212.835206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2212.924369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2213.879181] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2213.922809] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2213.926194] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:10048 to 0x280000400:10113 [ 2216.685463] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2216.688955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2216.727088] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2217.570956] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2218.864660] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:11857 to 0x2c0000400:11905 [ 2219.864771] Lustre: lustre-OST0001: deleting orphan objects from 0x0:15604 to 0x0:15681 [ 2219.865091] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17750 to 0x0:17825 [ 2225.118973] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2225.993200] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2232.627811] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 04:00:51 (1722240051) [ 2416.308708] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 04:03:55 (1722240235) [ 2419.542645] Lustre: *** cfs_fail_loc=327, val=0*** [ 2433.947661] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 04:04:13 (1722240253) [ 2435.583942] Lustre: *** cfs_fail_loc=328, val=0*** [ 2435.584942] Lustre: Skipped 512 previous similar messages [ 2437.159541] Lustre: *** cfs_fail_loc=328, val=0*** [ 2437.161550] Lustre: Skipped 252 previous similar messages [ 2439.171023] Lustre: *** cfs_fail_loc=328, val=0*** [ 2439.173118] Lustre: Skipped 1 previous similar message [ 2443.421315] Lustre: *** cfs_fail_loc=328, val=0*** [ 2443.422584] Lustre: Skipped 125 previous similar messages [ 2451.719309] Lustre: *** cfs_fail_loc=328, val=0*** [ 2451.722835] Lustre: Skipped 93 previous similar messages [ 2455.976341] LustreError: 17793:0:(ldlm_resource.c:133:seq_watermark_write()) Failed to set lock_reclaim_threshold_mb, rc = -22. [ 2460.030002] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [ 2460.415102] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [ 2460.821748] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 04:04:40 (1722240280) [ 2468.573085] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 04:04:47 (1722240287) [ 2477.278581] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 04:04:56 (1722240296) [ 2488.254035] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 04:05:07 (1722240307) [ 2498.569557] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 04:05:17 (1722240317) [ 2508.105047] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 04:05:27 (1722240327) [ 2511.635143] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 04:05:30 (1722240330) [ 2525.867569] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 04:05:45 (1722240345) [ 2538.512764] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 04:05:57 (1722240357) [ 2547.440877] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 04:06:06 (1722240366) [ 2552.472228] bash (26228): drop_caches: 1 [ 2556.602448] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 04:06:15 (1722240375) [ 2558.906474] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 04:06:18 (1722240378) [ 2560.940252] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 04:06:20 (1722240380) [ 2562.962589] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 04:06:21 (1722240381) [ 2565.534513] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 04:06:24 (1722240384) [ 2565.912171] LustreError: 26776:0:(fld_handler.c:263:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 2568.932439] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 04:06:27 (1722240387) [ 2569.289153] LustreError: 19069:0:(fld_handler.c:263:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 2569.294675] LustreError: 19069:0:(fld_handler.c:263:fld_server_lookup()) Skipped 3 previous similar messages [ 2571.946845] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 04:06:31 (1722240391) [ 2574.386917] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 04:06:33 (1722240393) [ 2577.068246] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 04:06:36 (1722240396) [ 2578.777593] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 04:06:37 (1722240397) [ 2580.925819] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 04:06:39 (1722240399) [ 2661.119313] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 04:08:00 (1722240480) [ 2664.792850] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 04:08:04 (1722240484) [ 2668.071513] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 04:08:07 (1722240487) [ 2671.473240] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 04:08:10 (1722240490) [ 2675.255625] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 04:08:14 (1722240494) [ 2689.512828] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 04:08:28 (1722240508) [ 2708.419272] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 04:08:47 (1722240527) [ 2728.831827] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 04:09:07 (1722240547) [ 2748.532557] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 04:09:27 (1722240567) [ 2751.645573] Lustre: DEBUG MARKER: Turn on read and write cache [ 2753.078765] Lustre: DEBUG MARKER: Write data and read it back. [ 2753.620953] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 2755.052750] Lustre: DEBUG MARKER: cache hits: before: 65581, after: 65584 [ 2755.618440] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 2756.638485] Lustre: DEBUG MARKER: cache hits:: before: 65584, after: 65587 [ 2757.180323] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [ 2758.769402] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 2760.312015] Lustre: DEBUG MARKER: cache hits:: before: 65587, after: 65590 [ 2760.761506] Lustre: DEBUG MARKER: Write data and read it back. [ 2761.212855] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 2762.729204] Lustre: DEBUG MARKER: cache hits:: before: 65590, after: 65593 [ 2763.074564] Lustre: DEBUG MARKER: Turn off read and write cache [ 2764.305458] Lustre: DEBUG MARKER: Write data and read it back [ 2764.908492] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 2766.441735] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65593 [ 2766.860200] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [ 2768.387463] Lustre: DEBUG MARKER: Write data and read it back [ 2768.954852] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 2770.542055] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65593 [ 2771.110245] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 2772.720966] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65596 [ 2775.835784] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 04:09:54 (1722240594) [ 2776.906658] Lustre: lustre-MDD0000: changelog on [ 2777.929150] Lustre: lustre-MDD0001: changelog on [ 2783.476012] Lustre: Failing over lustre-MDT0000 [ 2783.520361] LustreError: 10311:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [ 2783.554959] Lustre: server umount lustre-MDT0000 complete [ 2784.376235] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2785.034574] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2785.039897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2785.770723] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2785.777257] Lustre: Skipped 2 previous similar messages [ 2786.680313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.722246] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.722425] LustreError: 166-1: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2786.722551] LustreError: 11-0: MGC192.168.204.115@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 2786.722570] LustreError: 10908:0:(layout.c:2166:__req_capsule_get()) @@@ Wrong buffer for field 'mgs_target_info' (1 of 1) in format 'MGS_TARGET_REG', 0 vs. 4544 (server) req@ffff8800a690aac0 x1805899546282560/t0(0) o253->MGC192.168.204.115@tcp@0@lo:26/25 lens 4768/224 e 0 to 0 dl 1722240613 ref 1 fl Complete:RSQU/0/0 rc -108/-107 job:'mount.lustre.0' [ 2786.722680] Lustre: Evicted from MGS (at 192.168.204.115@tcp) after server handle changed from 0x7daa546d84e496b5 to 0x7daa546d84ed4dee [ 2786.723182] Lustre: MGC192.168.204.115@tcp: Connection restored to 192.168.204.115@tcp (at 0@lo) [ 2786.749406] LustreError: Skipped 8 previous similar messages [ 2786.785215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.793476] Lustre: lustre-MDD0000: changelog on [ 2786.796940] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2787.673407] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2789.374962] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2791.788431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2791.834635] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2791.858569] Lustre: lustre-OST0000: deleting orphan objects from 0x0:18650 to 0x0:18689 [ 2791.861366] Lustre: lustre-OST0001: deleting orphan objects from 0x0:16513 to 0x0:16545 [ 2792.156546] Lustre: lustre-MDD0000: changelog off [ 2792.790299] Lustre: lustre-MDD0001: changelog off [ 2797.826222] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 04:10:16 (1722240616) [ 2798.991459] Lustre: lustre-MDD0000: changelog on [ 2803.303360] Lustre: lustre-MDD0001: changelog off [ 2805.250222] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 04:10:24 (1722240624) [ 2806.352277] Lustre: lustre-MDD0000: changelog on [ 2806.354583] Lustre: Skipped 1 previous similar message [ 2810.884498] Lustre: lustre-MDD0001: changelog off [ 2810.886713] Lustre: Skipped 1 previous similar message [ 2812.622023] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 04:10:31 (1722240631) [ 2817.005583] Lustre: lustre-MDD0001: changelog off [ 2817.006894] Lustre: Skipped 1 previous similar message [ 2818.377353] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 04:10:37 (1722240637) [ 2819.044806] Lustre: lustre-MDD0000: changelog on [ 2819.045776] Lustre: Skipped 3 previous similar messages [ 2824.853607] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 04:10:43 (1722240643) [ 2829.368470] Lustre: DEBUG MARKER: 1722240648: creating first files [ 2844.563948] Lustre: *** cfs_fail_loc=1313, val=0*** [ 2844.566773] Lustre: 324:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 2844.573246] Lustre: 18861:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl7 idle for 16s with 4 unprocessed records [ 2847.024863] Lustre: DEBUG MARKER: sanity test_160f: @@@@@@ FAIL: mds2: User cl7 still registered