[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] mapped at [ffffffffff200b50] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2f2000-0xbffcffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5970 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 239638947 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffd9fff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043832 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782298 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027499 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820316k/5224448k available (8172k kernel code, 1049120k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.374639] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.377420] pid_max: default: 32768 minimum: 301 [ 0.379013] Security Framework initialized [ 0.380500] SELinux: Initializing. [ 0.381595] SELinux: Starting in permissive mode [ 0.383061] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.386266] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.388396] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.389981] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.391920] Initializing cgroup subsys memory [ 0.392758] Initializing cgroup subsys devices [ 0.393854] Initializing cgroup subsys freezer [ 0.394831] Initializing cgroup subsys net_cls [ 0.395738] Initializing cgroup subsys blkio [ 0.396605] Initializing cgroup subsys perf_event [ 0.397608] Initializing cgroup subsys hugetlb [ 0.398893] Initializing cgroup subsys pids [ 0.399918] Initializing cgroup subsys net_prio [ 0.401058] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.403593] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.404907] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.406169] tlb_flushall_shift: 6 [ 0.407070] FEATURE SPEC_CTRL Present [ 0.407901] FEATURE IBPB_SUPPORT Present [ 0.408756] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.410393] Spectre V2 : Vulnerable [ 0.411104] Speculative Store Bypass: Vulnerable [ 0.412832] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.419386] ACPI: Core revision 20130517 [ 0.421494] ACPI: All ACPI Tables successfully acquired [ 0.422693] ftrace: allocating 30294 entries in 119 pages [ 0.471617] Enabling x2apic [ 0.472174] Enabled x2apic [ 0.473026] Switched APIC routing to physical x2apic. [ 0.475245] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.476511] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.478610] TSC deadline timer enabled [ 0.478765] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.480664] ... version: 2 [ 0.481427] ... bit width: 48 [ 0.482161] ... generic registers: 4 [ 0.483099] ... value mask: 0000ffffffffffff [ 0.484226] ... max period: 00007fffffffffff [ 0.485323] ... fixed-purpose events: 3 [ 0.486025] ... event mask: 000000070000000f [ 0.487091] KVM setup paravirtual spinlock [ 0.490892] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.493932] KVM setup async PF for cpu 1 [ 0.495082] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.497199] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.500616] KVM setup async PF for cpu 2 [ 0.489715] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.501137] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.504195] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.506155] Brought up 4 CPUs [ 0.506199] KVM setup async PF for cpu 3 [ 0.506205] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.508417] smpboot: Max logical packages: 1 [ 0.509182] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.512437] devtmpfs: initialized [ 0.513957] x86/mm: Memory block size: 128MB [ 0.517762] EVM: security.selinux [ 0.518522] EVM: security.ima [ 0.519058] EVM: security.capability [ 0.521948] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.524225] NET: Registered protocol family 16 [ 0.525339] cpuidle: using governor haltpoll [ 0.526570] ACPI: bus type PCI registered [ 0.527319] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.528668] PCI: Using configuration type 1 for base access [ 0.529768] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.536687] ACPI: Added _OSI(Module Device) [ 0.537559] ACPI: Added _OSI(Processor Device) [ 0.538464] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.539384] ACPI: Added _OSI(Processor Aggregator Device) [ 0.540391] ACPI: Added _OSI(Linux-Dell-Video) [ 0.541976] ACPI: EC: Look up EC in DSDT [ 0.543833] ACPI: Interpreter enabled [ 0.544630] ACPI: (supports S0 S3 S4 S5) [ 0.545419] ACPI: Using IOAPIC for interrupt routing [ 0.546420] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.548237] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.553502] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.554796] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.556178] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.557482] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.560491] acpiphp: Slot [2] registered [ 0.561587] acpiphp: Slot [3] registered [ 0.563469] acpiphp: Slot [4] registered [ 0.565317] acpiphp: Slot [5] registered [ 0.566792] acpiphp: Slot [6] registered [ 0.567626] acpiphp: Slot [7] registered [ 0.568447] acpiphp: Slot [8] registered [ 0.569375] acpiphp: Slot [9] registered [ 0.570208] acpiphp: Slot [10] registered [ 0.571068] acpiphp: Slot [11] registered [ 0.571899] acpiphp: Slot [12] registered [ 0.572740] acpiphp: Slot [13] registered [ 0.573715] acpiphp: Slot [14] registered [ 0.574713] acpiphp: Slot [15] registered [ 0.575658] acpiphp: Slot [16] registered [ 0.576669] acpiphp: Slot [17] registered [ 0.577605] acpiphp: Slot [18] registered [ 0.578480] acpiphp: Slot [19] registered [ 0.579962] acpiphp: Slot [20] registered [ 0.581615] acpiphp: Slot [21] registered [ 0.583272] acpiphp: Slot [22] registered [ 0.584853] acpiphp: Slot [23] registered [ 0.586485] acpiphp: Slot [24] registered [ 0.587906] acpiphp: Slot [25] registered [ 0.589062] acpiphp: Slot [26] registered [ 0.589927] acpiphp: Slot [27] registered [ 0.590798] acpiphp: Slot [28] registered [ 0.591669] acpiphp: Slot [29] registered [ 0.592560] acpiphp: Slot [30] registered [ 0.593472] acpiphp: Slot [31] registered [ 0.594642] PCI host bridge to bus 0000:00 [ 0.596019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.598317] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.599726] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.601449] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.603084] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.605119] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.606534] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.607224] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.608035] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.611621] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.613277] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.615409] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.616900] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.618651] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.620452] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.620945] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.622675] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.624674] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.625738] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.631486] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.633107] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.634117] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.637900] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.640765] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.646602] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.650776] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.652658] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.654543] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.664507] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.668217] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.669626] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.670895] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.672834] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.674130] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.675886] vgaarb: loaded [ 0.676592] SCSI subsystem initialized [ 0.677371] ACPI: bus type USB registered [ 0.678189] usbcore: registered new interface driver usbfs [ 0.679258] usbcore: registered new interface driver hub [ 0.680318] usbcore: registered new device driver usb [ 0.681502] PCI: Using ACPI for IRQ routing [ 0.682292] PCI: pci_cache_line_size set to 64 bytes [ 0.682424] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.682427] e820: reserve RAM buffer [mem 0xbffda000-0xbfffffff] [ 0.682429] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 0.682663] NetLabel: Initializing [ 0.683314] NetLabel: domain hash size = 128 [ 0.684082] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.685035] NetLabel: unlabeled traffic allowed by default [ 0.686295] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.687266] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.691586] amd_nb: Cannot enumerate AMD northbridges [ 0.692639] Switched to clocksource kvm-clock [ 0.705368] pnp: PnP ACPI init [ 0.706344] ACPI: bus type PNP registered [ 0.707900] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.708002] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.708058] pnp 00:02: [dma 2] [ 0.708104] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.708196] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.708299] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.708400] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.708830] pnp: PnP ACPI: found 6 devices [ 0.710253] ACPI: bus type PNP unregistered [ 0.721197] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.721202] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.721205] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.721208] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.721212] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 0.721278] NET: Registered protocol family 2 [ 0.723467] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 0.726194] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 0.729342] TCP: Hash tables configured (established 32768 bind 32768) [ 0.731601] TCP: reno registered [ 0.732838] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 0.734983] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 0.737349] NET: Registered protocol family 1 [ 0.739343] RPC: Registered named UNIX socket transport module. [ 0.741456] RPC: Registered udp transport module. [ 0.743128] RPC: Registered tcp transport module. [ 0.744890] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.747270] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.749820] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.751775] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.753106] PCI: CLS 0 bytes, default 64 [ 0.753915] Unpacking initramfs... [ 2.019464] debug: unmapping init [mem 0xffff8800bc2f2000-0xffff8800bffcffff] [ 2.023140] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.024489] software IO TLB [mem 0xb82f2000-0xbc2f2000] (64MB) mapped at [ffff8800b82f2000-ffff8800bc2f1fff] [ 2.027101] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.028799] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.029921] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.031118] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.033943] cryptomgr_test (51) used greatest stack depth: 14128 bytes left [ 2.034319] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.034367] Initialise system trusted keyring [ 2.070568] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.073543] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.081871] zpool: loaded [ 2.083139] zbud: loaded [ 2.084843] VFS: Disk quotas dquot_6.6.0 [ 2.086752] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.090684] NFS: Registering the id_resolver key type [ 2.092587] Key type id_resolver registered [ 2.094492] Key type id_legacy registered [ 2.096149] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.099490] Key type big_key registered [ 2.101222] SELinux: Registering netfilter hooks [ 2.103058] cryptomgr_test (57) used greatest stack depth: 14048 bytes left [ 2.105664] cryptomgr_test (58) used greatest stack depth: 13968 bytes left [ 2.108402] NET: Registered protocol family 38 [ 2.109946] Key type asymmetric registered [ 2.111295] Asymmetric key parser 'x509' registered [ 2.113040] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.115589] io scheduler noop registered [ 2.116862] io scheduler deadline registered (default) [ 2.118595] io scheduler cfq registered [ 2.119804] io scheduler mq-deadline registered [ 2.121246] io scheduler kyber registered [ 2.124920] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.126840] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.129073] intel_idle: does not run on family 6 model 62 [ 2.129229] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.131503] ACPI: Power Button [PWRF] [ 2.133167] GHES: HEST is not enabled! [ 2.207722] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.287220] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.392699] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.423733] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 2.454944] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 2.458435] Non-volatile memory driver v1.3 [ 2.460084] Linux agpgart interface v0.103 [ 2.461961] crash memory driver: version 1.1 [ 2.464282] nbd: registered device at major 43 [ 2.469708] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 2.469757] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 2.469808] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 2.469848] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 2.469886] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 2.476890] virtio_blk virtio1: [vda] 116096 512-byte logical blocks (59.4 MB/56.6 MiB) [ 2.482651] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 2.482685] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 2.482719] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 2.482744] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 2.482768] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 2.489664] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.493973] rdac: device handler registered [ 2.495769] hp_sw: device handler registered [ 2.497478] emc: device handler registered [ 2.499163] libphy: Fixed MDIO Bus: probed [ 2.502109] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 2.504391] ehci-pci: EHCI PCI platform driver [ 2.505935] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 2.508072] ohci-pci: OHCI PCI platform driver [ 2.509648] uhci_hcd: USB Universal Host Controller Interface driver [ 2.511968] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.515856] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.517544] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.519726] mousedev: PS/2 mouse device common for all mice [ 2.522651] rtc_cmos 00:05: RTC can wake from S4 [ 2.523287] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.530200] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 2.533002] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.536958] hidraw: raw HID events driver (C) Jiri Kosina [ 2.539062] usbcore: registered new interface driver usbhid [ 2.540918] usbhid: USB HID core driver [ 2.542301] drop_monitor: Initializing network drop monitor service [ 2.544435] Netfilter messages via NETLINK v0.30. [ 2.546088] TCP: cubic registered [ 2.547192] Initializing XFRM netlink socket [ 2.549047] NET: Registered protocol family 10 [ 2.551289] NET: Registered protocol family 17 [ 2.552893] Key type dns_resolver registered [ 2.554991] mce: Using 10 MCE banks [ 2.556865] Loading compiled-in X.509 certificates [ 2.559204] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 2.561681] registered taskstats version 1 [ 2.566368] modprobe (71) used greatest stack depth: 13456 bytes left [ 2.571375] Key type trusted registered [ 2.576972] Key type encrypted registered [ 2.578560] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 2.582365] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 2.586498] rtc_cmos 00:05: setting system clock to 2025-06-11 01:29:50 UTC (1749605390) [ 2.590106] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 2.592841] Write protecting the kernel read-only data: 12288k [ 2.595358] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 2.597953] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 2.606424] random: systemd: uninitialized urandom read (16 bytes read) [ 2.609119] random: systemd: uninitialized urandom read (16 bytes read) [ 2.610533] random: systemd: uninitialized urandom read (16 bytes read) [ 2.613618] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 2.618668] systemd[1]: Detected virtualization kvm. [ 2.619729] systemd[1]: Detected architecture x86-64. [ 2.620854] systemd[1]: Running in initial RAM disk. [ 2.624582] systemd[1]: No hostname configured. [ 2.626087] systemd[1]: Set hostname to . [ 2.627229] random: systemd: uninitialized urandom read (16 bytes read) [ 2.629741] systemd[1]: Initializing machine ID from random generator. [ 2.671871] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 2.675405] random: systemd: uninitialized urandom read (16 bytes read) [ 2.677970] random: systemd: uninitialized urandom read (16 bytes read) [ 2.680516] random: systemd: uninitialized urandom read (16 bytes read) [ 2.683074] random: systemd: uninitialized urandom read (16 bytes read) [ 2.687023] random: systemd: uninitialized urandom read (16 bytes read) [ 2.689721] random: systemd: uninitialized urandom read (16 bytes read) [ 2.701417] systemd[1]: Reached target Local File Systems. [ 2.706738] systemd[1]: Created slice Root Slice. [ 2.710950] systemd[1]: Created slice System Slice. [ 2.715125] systemd[1]: Listening on udev Control Socket. [ 2.719797] systemd[1]: Listening on Journal Socket. [ 2.725614] systemd[1]: Starting Journal Service... [ 2.730807] systemd[1]: Starting dracut cmdline hook... [ 2.737010] systemd[1]: Starting Setup Virtual Console... [ 2.741105] systemd[1]: Listening on udev Kernel Socket. [ 2.745367] systemd[1]: Reached target Sockets. [ 2.748960] systemd[1]: Reached target Slices. [ 2.752725] systemd[1]: Reached target Swap. [ 2.757374] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 2.762483] systemd[1]: Starting Load Kernel Modules... [ 2.765512] systemd[1]: Reached target Timers. [ 2.770512] systemd[1]: Started Journal Service. [ 3.033755] tsc: Refined TSC clocksource calibration: 2399.961 MHz [ 3.128939] random: fast init done [ 3.242358] libata version 3.00 loaded. [ 3.246219] ata_piix 0000:00:01.1: version 2.13 [ 3.248024] scsi host0: ata_piix [ 3.256170] scsi host1: ata_piix [ 3.256966] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 3.258801] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 3.338877] ip (320) used greatest stack depth: 13080 bytes left [ 3.369266] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 3.390430] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 3.392714] ip (343) used greatest stack depth: 12464 bytes left [ 3.503739] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 5.547599] ip (381) used greatest stack depth: 12432 bytes left [ 6.357909] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 6.708994] systemd-journald[100]: Received SIGTERM from PID 1 (systemd). [ 6.887104] SELinux: Disabled at runtime. [ 6.888680] SELinux: Unregistering netfilter hooks [ 6.941729] ip_tables: (C) 2000-2006 Netfilter Core Team [ 6.944549] systemd[1]: Inserted module 'ip_tables' [ 7.199388] systemd-journald[563]: Received request to flush runtime journal from PID 1 [ 7.302865] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 7.311599] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 7.333901] cryptd: max_cpu_qlen set to 1000 [ 7.361532] AVX version of gcm_enc/dec engaged. [ 7.362727] AES CTR mode by8 optimization enabled [ 7.405570] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 7.422991] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 7.431368] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 7.434802] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 7.518216] EDAC MC: Ver: 3.0.0 [ 7.524382] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 7.524390] EDAC sbridge: Ver: 1.1.2 [ 8.958251] mount.nfs (763) used greatest stack depth: 10704 bytes left [ 30.617420] libcfs: loading out-of-tree module taints kernel. [ 30.618859] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 30.751024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 30.756518] alg: No test for adler32 (adler32-zlib) [ 31.644417] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 31.820342] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 31.822789] LNet: Accept secure, port 988 [ 33.367829] Key type lgssc registered [ 33.621392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.340448] Lustre: Mounted lustre-client [ 85.229563] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 97.677335] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing check_logdir /tmp/testlogs/ [ 98.516068] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing yml_node [ 99.492942] Lustre: DEBUG MARKER: Client: 2.16.56.1 [ 100.068153] Lustre: DEBUG MARKER: MDS: 2.16.56.1 [ 100.694119] Lustre: DEBUG MARKER: OSS: 2.16.56.1 [ 101.064454] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Tue Jun 10 21:31:28 EDT 2025 [ 105.284291] Lustre: DEBUG MARKER: - need mds1 <= 2.14.55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 105.621680] Lustre: DEBUG MARKER: - need mds1 < v2_14_55-100-g8a84c7f9c7 for LU-14927, skip 0f [ 105.960779] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42c 42b 118c 118d 407 119i 411b [ 106.318235] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 [ 106.693169] Lustre: DEBUG MARKER: === sanity: start setup 21:31:34 (1749605494) === [ 107.457292] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing check_config_client /mnt/lustre [ 108.365852] Lustre: lustre-OST0000-osc-ffff8800b3ba7000: disconnect after 24s idle [ 112.243151] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 115.009165] Lustre: DEBUG MARKER: === sanity: finish setup 21:31:42 (1749605502) === [ 116.402524] rm (11408) used greatest stack depth: 10464 bytes left [ 117.449286] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 21:31:45 (1749605505) [ 121.584734] lfs (11892) used greatest stack depth: 10432 bytes left [ 129.272622] random: crng init done [ 130.652719] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 21:31:58 (1749605518) [ 132.359749] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 21:31:59 (1749605519) [ 134.091383] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 21:32:01 (1749605521) [ 135.734020] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 21:32:03 (1749605523) [ 137.386088] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 21:32:04 (1749605524) [ 139.054859] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 21:32:06 (1749605526) [ 140.660926] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 21:32:08 (1749605528) [ 142.277380] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 21:32:09 (1749605529) [ 143.888370] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 21:32:11 (1749605531) [ 148.679612] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [ 149.079162] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.12417 [ 149.722319] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [ 150.100695] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.29025 [ 150.752852] Lustre: DEBUG MARKER: Test: lfs mkdir -i 1 /mnt/lustre/d205a.sanity.remote [ 151.129755] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.16662 [ 151.808220] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [ 152.205882] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.17482 [ 152.867843] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [ 153.259415] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.30312 [ 153.910665] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [ 154.298721] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.12550 [ 154.946111] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 155.363241] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.13377 [ 156.268824] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [ 156.671146] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.8230 [ 157.404524] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [ 157.808886] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.15699 [ 158.503527] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [ 158.889385] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.6048 [ 159.734072] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [ 160.121067] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.11788 [ 160.762409] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [ 161.135155] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.11176 [ 164.546138] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 164.900128] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg434-client.v [ 165.545110] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 165.938308] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg434-client.E [ 166.619976] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 167.023505] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.oleg434-client.v [ 172.893809] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 21:32:40 (1749605560) [ 176.001560] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 21:32:43 (1749605563) [ 177.606033] Lustre: DEBUG MARKER: == sanity test 205d: verify the format of some stats files ========================================================== 21:32:45 (1749605565) [ 177.918513] lfs (25202) used greatest stack depth: 10416 bytes left [ 181.703572] Lustre: DEBUG MARKER: == sanity test 205e: verify the output of lljobstat ====== 21:32:49 (1749605569) [ 186.623931] Lustre: DEBUG MARKER: == sanity test 205f: verify qos_ost_weights YAML format == 21:32:54 (1749605574) [ 188.913515] Lustre: DEBUG MARKER: == sanity test 205g: stress test for job_stats procfile == 21:32:56 (1749605576) [ 281.544101] Lustre: DEBUG MARKER: == sanity test 205h: check jobid xattr is stored correctly ========================================================== 21:34:29 (1749605669) [ 284.617277] Lustre: DEBUG MARKER: == sanity test 205i: check job_xattr parameter accepts and rejects values correctly ========================================================== 21:34:32 (1749605672) [ 288.083395] Lustre: DEBUG MARKER: == sanity test 205k: Verify '?' operator on job stats ==== 21:34:35 (1749605675) [ 290.622157] Lustre: DEBUG MARKER: == sanity test 205l: Verify job stats can scale ========== 21:34:38 (1749605678) [ 326.967445] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 21:35:14 (1749605714) [ 327.036079] Lustre: *** cfs_fail_loc=1403, val=1*** [ 327.038077] LustreError: 29393:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000402:0x11:0x0]: rc = -5 [ 327.042404] LustreError: 29393:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5 [ 328.563199] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 21:35:16 (1749605716) [ 330.196773] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 21:35:17 (1749605717) [ 331.891195] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 21:35:19 (1749605719) [ 338.735138] Lustre: lustre-MDT0000-mdc-ffff8800b3ba7000: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 353.758013] Lustre: lustre-OST0000-osc-ffff8800b3ba7000: disconnect after 20s idle [ 353.759453] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail [ 353.761479] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xa698cb9111c4f3f7 to 0xa698cb9111cae734 [ 353.762106] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 192.168.204.134@tcp) [ 353.767672] LustreError: 2005:0:(client.c:3393:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88012c7c2a00 x1834594257884032/t4294983740(4294983740) o101->lustre-MDT0000-mdc-ffff8800b3ba7000@192.168.204.134@tcp:12/10 lens 576/608 e 0 to 0 dl 1749605757 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 [ 353.777354] Lustre: Skipped 1 previous similar message [ 355.747332] Lustre: lustre-MDT0000-mdc-ffff8800b3ba7000: Connection restored to (at 192.168.204.134@tcp) [ 356.519511] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 356.899260] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 363.791864] Lustre: lustre-MDT0000-mdc-ffff8800b3ba7000: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 373.807243] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail [ 373.813801] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xa698cb9111cae734 to 0xa698cb9111caeb32 [ 373.817764] LustreError: 2005:0:(client.c:3393:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800a84b6d80 x1834594257891200/t8589934596(8589934596) o101->lustre-MDT0000-mdc-ffff8800b3ba7000@192.168.204.134@tcp:12/10 lens 584/608 e 0 to 0 dl 1749605777 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 [ 373.826818] LustreError: 2005:0:(client.c:3393:ptlrpc_replay_interpret()) Skipped 1 previous similar message [ 373.827212] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 192.168.204.134@tcp) [ 377.925321] Lustre: lustre-MDT0000-mdc-ffff8800b3ba7000: Connection restored to (at 192.168.204.134@tcp) [ 378.698562] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 379.086364] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 381.920596] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 21:36:09 (1749605769) [ 387.091006] bash (1069): drop_caches: 3 [ 388.918364] bash (1069): drop_caches: 3 [ 390.696685] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 21:36:18 (1749605778) [ 394.249829] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 21:36:21 (1749605781) [ 398.082258] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 21:36:25 (1749605785) [ 398.135387] LustreError: 2009:0:(osc_request.c:3105:osc_enqueue_interpret()) cfs_fail_timeout id 40f sleeping for 10000ms [ 408.153748] LustreError: 2009:0:(osc_request.c:3105:osc_enqueue_interpret()) cfs_fail_timeout id 40f awake [ 408.165917] cat (3684) used greatest stack depth: 10240 bytes left [ 409.610833] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 21:36:37 (1749605797) [ 417.289802] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 21:36:44 (1749605804) [ 418.952825] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 21:36:46 (1749605806) [ 423.995719] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with embedded hyphen ('-') ========================================================== 21:36:51 (1749605811) [ 426.103738] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 21:36:53 (1749605813) [ 426.481230] Lustre: DEBUG MARKER: creating a 10 Mb file [ 439.761385] Lustre: DEBUG MARKER: starting reads [ 440.132228] Lustre: DEBUG MARKER: truncating the file [ 440.612383] Lustre: DEBUG MARKER: killing dd [ 440.956976] Lustre: DEBUG MARKER: removing the temporary file [ 442.451729] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 21:37:10 (1749605830) [ 442.512002] Lustre: *** cfs_fail_loc=411, val=0*** [ 443.999344] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 21:37:11 (1749605831) [ 456.097067] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 21:37:23 (1749605843) [ 458.202498] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 21:37:25 (1749605845) [ 459.938722] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 21:37:27 (1749605847) [ 461.688243] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 21:37:29 (1749605849) [ 463.403874] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 21:37:30 (1749605850) [ 463.481554] Lustre: *** cfs_fail_loc=508, val=2147483648*** [ 463.483361] LustreError: 1999:0:(events.c:193:client_bulk_callback()) event type 1, status -5, desc ffff8800b5895400 [ 463.487093] Lustre: 2009:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1749605851/real 1749605851] req@ffff8800a8190a80 x1834594258526080/t0(0) o4->lustre-OST0001-osc-ffff8800b3ba7000@192.168.204.134@tcp:6/4 lens 488/448 e 0 to 1 dl 1749605867 ref 2 fl Rpc:eXQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 [ 463.497027] Lustre: lustre-OST0001-osc-ffff8800b3ba7000: Connection to lustre-OST0001 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 463.506190] Lustre: lustre-OST0001-osc-ffff8800b3ba7000: Connection restored to (at 192.168.204.134@tcp) [ 467.125692] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 21:37:34 (1749605854) [ 473.527674] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 21:37:41 (1749605861) [ 483.533810] Lustre: 2007:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1749605866/real 1749605866] req@ffff8800a836ea00 x1834594258540544/t0(0) o4->lustre-OST0000-osc-ffff8800b3ba7000@192.168.204.134@tcp:6/4 lens 488/448 e 0 to 1 dl 1749605871 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 [ 483.542145] Lustre: lustre-OST0000-osc-ffff8800b3ba7000: Connection to lustre-OST0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 483.552496] Lustre: lustre-OST0000-osc-ffff8800b3ba7000: Connection restored to (at 192.168.204.134@tcp) [ 487.555953] Lustre: lustre-OST0001-osc-ffff8800b3ba7000: disconnect after 20s idle [ 487.558831] Lustre: Skipped 1 previous similar message [ 489.575774] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 21:37:57 (1749605877) [ 510.826788] Lustre: 2009:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1749605878/real 1749605878] req@ffff8801369c9500 x1834594258553472/t0(0) o3->lustre-OST0000-osc-ffff8800b3ba7000@192.168.204.134@tcp:6/4 lens 488/440 e 0 to 1 dl 1749605898 ref 2 fl Bulk:RXMQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 [ 510.833767] Lustre: lustre-OST0000-osc-ffff8800b3ba7000: Connection to lustre-OST0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 510.837797] LustreError: 2009:0:(client.c:2320:ptlrpc_check_set()) @@@ bulk transfer failed 0/1048576/0 req@ffff8801369c9500 x1834594258553472/t0(0) o3->lustre-OST0000-osc-ffff8800b3ba7000@192.168.204.134@tcp:6/4 lens 488/440 e 0 to 1 dl 1749605898 ref 2 fl Bulk:ReXMQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 [ 510.843661] Lustre: lustre-OST0000-osc-ffff8800b3ba7000: Connection restored to (at 192.168.204.134@tcp) [ 510.848559] LustreError: 2009:0:(osc_request.c:2444:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8801369c9500 x1834594258553472/t0(0) o3->lustre-OST0000-osc-ffff8800b3ba7000@192.168.204.134@tcp:6/4 lens 488/440 e 0 to 1 dl 1749605898 ref 2 fl Interpret:ReXMQU/600/0 rc -5/0 job:'dd.0' uid:0 gid:0 projid:0 [ 514.068560] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [ 514.464072] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [ 514.872548] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 21:38:22 (1749605902) [ 516.632960] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 21:38:24 (1749605904) [ 518.421038] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 21:38:25 (1749605905) [ 518.598461] Lustre: Mounted lustre-client [ 518.649487] LustreError: 18559:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8800a6b52800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 518.656370] LustreError: 18559:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 518.680725] Lustre: Unmounted lustre-client [ 519.954578] LustreError: 19054:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff88012b42c800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 519.958067] LustreError: 19054:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 519.962674] LustreError: 19054:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 519.964360] LustreError: 19054:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 519.983746] Lustre: Unmounted lustre-client [ 520.355211] Lustre: DEBUG MARKER: == sanity test 226d: verify fid2path with -n and -fn option ========================================================== 21:38:27 (1749605907) [ 522.109410] Lustre: DEBUG MARKER: == sanity test 226e: Verify path2fid -0 option with newline and space ========================================================== 21:38:29 (1749605909) [ 523.687560] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 21:38:31 (1749605911) [ 525.222969] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 21:38:32 (1749605912) [ 525.857428] Lustre: *** cfs_fail_loc=1002, val=0*** [ 548.782035] Lustre: lustre-OST0001-osc-ffff8800b3ba7000: disconnect after 23s idle [ 573.462890] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 21:39:21 (1749605961) [ 574.114732] Lustre: *** cfs_fail_loc=1002, val=0*** [ 613.887456] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail [ 613.887564] Lustre: lustre-MDT0000-mdc-ffff8800b3ba7000: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 614.010083] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xa698cb9111caeb32 to 0xa698cb9111e226a7 [ 614.013742] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 192.168.204.134@tcp) [ 629.894545] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 21:40:17 (1749606017) [ 630.518373] Lustre: *** cfs_fail_loc=1002, val=0*** [ 638.925736] Lustre: lustre-OST0001-osc-ffff8800b3ba7000: disconnect after 21s idle [ 689.005710] Lustre: lustre-OST0001-osc-ffff8800b3ba7000: disconnect after 22s idle [ 706.090413] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 21:41:33 (1749606093) [ 707.908989] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 21:41:35 (1749606095) [ 709.949982] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 21:41:37 (1749606097) [ 724.009465] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 21:41:51 (1749606111) [ 728.500659] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [ 728.907928] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 21:41:56 (1749606116) [ 730.917196] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 21:41:58 (1749606118) [ 733.075838] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 21:42:00 (1749606120) [ 734.707377] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 21:42:02 (1749606122) [ 736.446363] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 21:42:04 (1749606124) [ 738.308870] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 21:42:05 (1749606125) [ 740.188155] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 21:42:07 (1749606127) [ 740.567127] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [ 741.059815] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 21:42:08 (1749606128) [ 768.697920] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 21:42:36 (1749606156) [ 770.150006] bash (1242): drop_caches: 3 [ 770.523735] bash (1242): drop_caches: 3 [ 772.450516] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 21:42:40 (1749606160) [ 774.479393] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 21:42:42 (1749606162) [ 780.948900] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 21:42:48 (1749606168) [ 791.496914] LustreError: 3893:0:(llite_lib.c:1872:ll_update_lsm_md()) lustre: [0x2000013a1:0x3498:0x0] dir layout mismatch: [ 791.499082] LustreError: 3893:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=1 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool= [ 791.502727] LustreError: 3893:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x200001b70:0x7d:0x0] [ 791.504660] LustreError: 3893:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=2 index=0 hash=crush:0x88000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=1 migrate_hash=crush:86000003 pool= [ 791.508537] LustreError: 3893:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -22 [ 800.827334] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 21:43:08 (1749606188) [ 819.358181] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 21:43:26 (1749606206) [ 821.977339] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 21:43:29 (1749606209) [ 825.108810] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 21:43:32 (1749606212) [ 827.712175] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 21:43:35 (1749606215) [ 828.124996] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [ 828.730046] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 21:43:36 (1749606216) [ 829.242775] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [ 829.742888] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 21:43:37 (1749606217) [ 832.553534] Lustre: DEBUG MARKER: == sanity test 230x: dir migration check space =========== 21:43:40 (1749606220) [ 855.870068] Lustre: DEBUG MARKER: == sanity test 230y: unlink dir with bad hash type ======= 21:44:03 (1749606243) [ 861.904888] Lustre: DEBUG MARKER: == sanity test 230z: resume dir migration with bad hash type ========================================================== 21:44:09 (1749606249) [ 875.753637] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 21:44:23 (1749606263) [ 879.722503] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 21:44:27 (1749606267) [ 898.234111] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 21:44:45 (1749606285) [ 898.558943] LustreError: lustre-OST0000-osc-ffff8800b3ba7000: operation ldlm_enqueue to node 192.168.204.134@tcp failed: rc = -12 [ 899.109137] LustreError: 13797:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8800b3ba7000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 899.111975] LustreError: 13797:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 899.116618] LustreError: 13797:0:(obd_class.h:479:obd_check_dev()) Device 6 not setup [ 899.118103] LustreError: 13797:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 899.145698] Lustre: Unmounted lustre-client [ 899.334336] Lustre: Mounted lustre-client [ 899.336258] Lustre: Skipped 1 previous similar message [ 903.822930] Lustre: lustre-OST0000-osc-ffff88012b11c000: Connection to lustre-OST0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 904.491882] Lustre: lustre-OST0000-osc-ffff88012b11c000: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp) [ 904.496137] Lustre: Skipped 1 previous similar message [ 906.502031] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 21:44:53 (1749606293) [ 907.460825] LustreError: 14759:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff88012b11c000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 907.466698] LustreError: 14759:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 907.475253] LustreError: 14759:0:(obd_class.h:479:obd_check_dev()) Device 6 not setup [ 907.479085] LustreError: 14759:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 907.514799] Lustre: Unmounted lustre-client [ 907.687961] Lustre: Mounted lustre-client [ 915.585818] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 21:45:03 (1749606303) [ 917.774757] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 21:45:05 (1749606305) [ 920.062397] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 21:45:07 (1749606307) [ 920.224717] Lustre: *** cfs_fail_loc=1405, val=0*** [ 922.614505] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 21:45:10 (1749606310) [ 926.699769] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 21:45:14 (1749606314) [ 929.397925] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 21:45:16 (1749606316) [ 931.643385] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 21:45:19 (1749606319) [ 964.409433] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 21:45:51 (1749606351) [ 968.711503] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 21:45:56 (1749606356) [ 973.558566] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 21:46:00 (1749606360) [ 973.969682] LustreError: 22513:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8800b5984000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 973.975227] LustreError: 22513:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 973.980801] LustreError: 22513:0:(obd_class.h:479:obd_check_dev()) Device 6 not setup [ 973.984725] LustreError: 22513:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 974.022826] Lustre: Unmounted lustre-client [ 974.523145] Lustre: Mounted lustre-client [ 979.760853] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 21:46:07 (1749606367) [ 1009.702477] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 21:46:37 (1749606397) [ 1021.031357] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 21:46:48 (1749606408) [ 1021.493990] LustreError: lustre-MDT0000-mdc-ffff8800b3ba1800: operation mds_readpage to node 192.168.204.134@tcp failed: rc = -12 [ 1024.045417] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 21:46:51 (1749606411) [ 1026.552674] Lustre: 1790:0:(file.c:3143:ll_get_grouplock()) lustre: group lock already exists with gid 97486 on [0x200001b73:0x5:0x0]: rc = -22 [ 1026.561868] Lustre: 1790:0:(file.c:3218:ll_put_grouplock()) lustre: no group lock held on [0x200001b73:0x5:0x0]: rc = -22 [ 1026.567822] Lustre: 1790:0:(file.c:3125:ll_get_grouplock()) lustre: group id for group lock on [0x200001b73:0x5:0x0] is 0: rc = -22 [ 1026.577602] Lustre: 1790:0:(file.c:3228:ll_put_grouplock()) lustre: group lock 4294967286 doesn't match current id 3543 on [0x200001b73:0x5:0x0]: rc = -22 [ 1130.662465] Lustre: 1790:0:(file.c:3218:ll_put_grouplock()) lustre: no group lock held on [0x200001b73:0xc:0x0]: rc = -22 [ 1130.681480] Lustre: 1790:0:(file.c:3125:ll_get_grouplock()) lustre: group id for group lock on [0x200001b73:0xc:0x0] is 0: rc = -22 [ 1133.500933] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 21:48:40 (1749606520) [ 1172.477544] sendfile_groupl (2580) used greatest stack depth: 9760 bytes left [ 1174.928475] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 21:49:22 (1749606562) [ 1177.622239] Lustre: DEBUG MARKER: == sanity test 245a: check mdc connection flag/data: multiple modify RPCs ========================================================== 21:49:24 (1749606564) [ 1180.111103] Lustre: DEBUG MARKER: == sanity test 245b: check osp connection flag/data: multiple modify RPCs ========================================================== 21:49:27 (1749606567) [ 1183.281924] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 21:49:30 (1749606570) [ 1183.458985] Lustre: Mounted lustre-client [ 1183.533740] LustreError: 5299:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8800a3a79800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1183.541242] LustreError: 5299:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 1183.550736] LustreError: 5299:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1183.555836] LustreError: 5299:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 1183.591816] Lustre: Unmounted lustre-client [ 1186.370373] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 21:49:33 (1749606573) [ 1186.523199] LustreError: 5990:0:(llite_lib.c:690:client_common_fill_super()) cannot mds_connect: rc = -2 [ 1186.562589] LustreError: 5990:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1189.056792] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 21:49:36 (1749606576) [ 1192.153410] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 21:49:39 (1749606579) [ 1195.179369] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 21:49:42 (1749606582) [ 1195.544896] LustreError: lustre-MDT0000-mdc-ffff8800a95fb000: operation mds_get_root to node 192.168.204.134@tcp failed: rc = -22 [ 1195.551395] LustreError: 8097:0:(llite_lib.c:690:client_common_fill_super()) cannot mds_connect: rc = -22 [ 1195.584990] LustreError: 8097:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 1197.704922] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 21:49:45 (1749606585) [ 1199.647750] Lustre: Mounted lustre-client [ 1199.649723] Lustre: Skipped 10 previous similar messages [ 1199.721531] LustreError: 8993:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff88012a63c000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1199.727787] LustreError: 8993:0:(lov_obd.c:784:lov_cleanup()) Skipped 21 previous similar messages [ 1199.735479] LustreError: 8993:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1199.739085] LustreError: 8993:0:(obd_class.h:479:obd_check_dev()) Skipped 103 previous similar messages [ 1199.761698] Lustre: Unmounted lustre-client [ 1199.762607] Lustre: Skipped 12 previous similar messages [ 1203.362570] Lustre: DEBUG MARKER: == sanity test 247g: striped directory submount revalidate ROOT from cache ========================================================== 21:49:50 (1749606590) [ 1207.128489] Lustre: DEBUG MARKER: == sanity test 247h: remote directory submount revalidate ROOT from cache ========================================================== 21:49:54 (1749606594) [ 1212.638434] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 21:50:00 (1749606600) [ 1250.637802] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 21:50:38 (1749606638) [ 1270.395915] Lustre: DEBUG MARKER: == sanity test 248c: verify whole file read behavior ===== 21:50:57 (1749606657) [ 1277.338454] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 21:51:04 (1749606664) [ 1279.905161] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 21:51:07 (1749606667) [ 1282.404278] Lustre: DEBUG MARKER: == sanity test 251a: Handling short read and write correctly ========================================================== 21:51:09 (1749606669) [ 1283.018096] Lustre: *** cfs_fail_loc=1407, val=0*** [ 1285.130238] Lustre: DEBUG MARKER: == sanity test 251b: short read restore offset correctly ========================================================== 21:51:12 (1749606672) [ 1285.184072] LustreError: 15620:0:(file.c:2464:do_file_read_iter()) cfs_fail_timeout id 1431 sleeping for 5000ms [ 1290.185881] LustreError: 15620:0:(file.c:2464:do_file_read_iter()) cfs_fail_timeout id 1431 awake [ 1292.511330] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 21:51:19 (1749606679) [ 1296.353285] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 21:51:23 (1749606683) [ 1365.752327] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 21:52:33 (1749606753) [ 1373.110566] Lustre: DEBUG MARKER: SKIP: sanity test_255a skipping excluded test 255a (base 255) [ 1373.610129] Lustre: DEBUG MARKER: SKIP: sanity test_255b skipping excluded test 255b (base 255) [ 1374.179259] Lustre: DEBUG MARKER: SKIP: sanity test_255c skipping excluded test 255c (base 255) [ 1374.692483] Lustre: DEBUG MARKER: SKIP: sanity test_256 skipping excluded test 256 [ 1375.239156] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 21:52:42 (1749606762) [ 1375.896511] LustreError: lustre-MDT0001-mdc-ffff8800b3ba1800: operation ldlm_enqueue to node 192.168.204.134@tcp failed: rc = -14 [ 1379.409165] Lustre: lustre-MDT0001-mdc-ffff8800b3ba1800: Connection to lustre-MDT0001 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1379.413577] Lustre: Skipped 1 previous similar message [ 1389.971057] Lustre: lustre-MDT0001-mdc-ffff8800b3ba1800: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp) [ 1389.976165] Lustre: Skipped 1 previous similar message [ 1394.604533] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 21:53:02 (1749606782) [ 1397.281092] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 21:53:04 (1749606784) [ 1399.874910] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 21:53:07 (1749606787) [ 1424.178567] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 21:53:31 (1749606811) [ 1424.255389] Lustre: *** cfs_fail_loc=806, val=0*** [ 1424.257306] Lustre: 23986:0:(mdc_request.c:913:mdc_close()) lustre-MDT0000-mdc-ffff8800b3ba1800: close of FID [0x200001b73:0x52:0x0] failed, file reference will be dropped when this client unmounts or is evicted [ 1424.263883] LustreError: 23986:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b3ba1800: inode [0x200001b73:0x52:0x0] mdc close failed: rc = -12 [ 1426.585240] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 21:53:33 (1749606813) [ 1432.021909] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 21:53:39 (1749606819) [ 1434.649985] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 21:53:41 (1749606821) [ 1437.236898] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 21:53:44 (1749606824) [ 1439.786840] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 21:53:47 (1749606827) [ 1443.161619] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 21:53:50 (1749606830) [ 1449.573184] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 21:53:56 (1749606836) [ 1459.192036] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 21:54:06 (1749606846) [ 1462.750975] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 21:54:10 (1749606850) [ 1465.097318] Lustre: DEBUG MARKER: == sanity test 270j: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 21:54:12 (1749606852) [ 1468.030018] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 21:54:15 (1749606855) [ 1470.593836] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 21:54:17 (1749606857) [ 1473.279222] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 21:54:20 (1749606860) [ 1476.019187] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 21:54:23 (1749606863) [ 1526.917608] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 21:55:14 (1749606914) [ 1529.779801] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 21:55:17 (1749606917) [ 1532.666974] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 21:55:19 (1749606919) [ 1533.758718] Lustre: *** cfs_fail_loc=314, val=0*** [ 1536.089906] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 21:55:23 (1749606923) [ 1538.242818] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 21:55:25 (1749606925) [ 1541.660784] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 21:55:29 (1749606929) [ 1544.648000] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 21:55:32 (1749606932) [ 1545.679301] LustreError: 6603:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b3ba1800: inode [0x240000409:0x80e:0x0] mdc close failed: rc = -22 [ 1548.062487] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 21:55:35 (1749606935) [ 1548.647854] LustreError: 7276:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b3ba1800: inode [0x200001b73:0x9d:0x0] mdc close failed: rc = -22 [ 1550.591807] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 21:55:38 (1749606938) [ 1551.088616] LustreError: 7936:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b3ba1800: inode [0x240000409:0x812:0x0] mdc close failed: rc = -22 [ 1553.014890] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 21:55:40 (1749606940) [ 1555.075565] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 21:55:42 (1749606942) [ 1560.366150] Lustre: DEBUG MARKER: == sanity test 273c: race writeback and object destroy === 21:55:47 (1749606947) [ 1563.717314] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 21:55:51 (1749606951) [ 1564.116163] LustreError: 22544:0:(ldlm_lockd.c:2902:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms [ 1566.121758] LustreError: 22544:0:(ldlm_lockd.c:2902:ldlm_bl_thread_blwi()) cfs_fail_timeout interrupted [ 1567.790097] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 21:55:55 (1749606955) [ 1571.762148] Lustre: lustre-OST0000-osc-ffff8800b3ba1800: Connection to lustre-OST0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1571.771598] Lustre: Skipped 1 previous similar message [ 1648.720358] Lustre: lustre-OST0000-osc-ffff8800b3ba1800: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp) [ 1648.722865] Lustre: Skipped 16 previous similar messages [ 1660.753008] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 21:57:28 (1749607048) [ 1663.235715] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 21:57:30 (1749607050) [ 1671.919453] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail [ 1671.927575] LustreError: 2005:0:(client.c:3393:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800a34ed180 x1834594312617088/t17179902394(17179902394) o101->lustre-MDT0000-mdc-ffff8800b3ba1800@192.168.204.134@tcp:12/10 lens 584/608 e 0 to 0 dl 1749607075 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 [ 1671.936414] LustreError: 2005:0:(client.c:3393:ptlrpc_replay_interpret()) Skipped 2 previous similar messages [ 1671.939589] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xa698cb911205d1f6 to 0xa698cb911210ab60 [ 1683.776049] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 21:57:51 (1749607071) [ 1684.069813] LustreError: 18388:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8800b3ba1800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1684.072749] LustreError: 18388:0:(lov_obd.c:784:lov_cleanup()) Skipped 23 previous similar messages [ 1684.076739] LustreError: 18388:0:(obd_class.h:479:obd_check_dev()) Device 6 not setup [ 1684.078342] LustreError: 18388:0:(obd_class.h:479:obd_check_dev()) Skipped 95 previous similar messages [ 1684.098328] Lustre: Unmounted lustre-client [ 1684.099331] Lustre: Skipped 11 previous similar messages [ 1694.544962] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail [ 1694.555581] LustreError: MGC192.168.204.134@tcp: Confguration from log lustre-client failed from MGS -5. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1694.557579] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xa698cb911210af7a to 0xa698cb911210b141 [ 1694.589883] LustreError: 18411:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -5 [ 1708.472567] Lustre: Mounted lustre-client [ 1708.474407] Lustre: Skipped 11 previous similar messages [ 1710.771747] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 21:58:18 (1749607098) [ 1713.613224] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 21:58:21 (1749607101) [ 1736.785472] Lustre: DEBUG MARKER: == sanity test 300c: chown [ 1854.930411] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 22:00:42 (1749607242) [ 1858.125333] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ========================================================== 22:00:45 (1749607245) [ 1860.694854] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ========================================================== 22:00:48 (1749607248) [ 1863.379977] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ========================================================== 22:00:50 (1749607250) [ 1868.464392] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory ========================================================== 22:00:55 (1749607255) [ 1873.282157] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory ========================================================== 22:01:00 (1749607260) [ 1873.868259] LustreError: 25094:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8800b3ee6000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1873.871527] LustreError: 25094:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 1873.875763] LustreError: 25094:0:(obd_class.h:479:obd_check_dev()) Device 6 not setup [ 1873.877249] LustreError: 25094:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 1873.901702] Lustre: Unmounted lustre-client [ 1873.902797] Lustre: Skipped 1 previous similar message [ 1874.032477] Lustre: Mounted lustre-client [ 1874.457410] Lustre: *** cfs_fail_loc=1901, val=99*** [ 1874.473069] Lustre: *** cfs_fail_loc=1901, val=99*** [ 1907.162697] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ============ 22:01:34 (1749607294) [ 1909.295406] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ======== 22:01:36 (1749607296) [ 1912.116718] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ========================================================== 22:01:39 (1749607299) [ 1915.243908] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS ========================================================== 22:01:42 (1749607302) [ 1915.773734] Lustre: DEBUG MARKER: SKIP: sanity test_300m Only for single MDT [ 1916.406850] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA ========================================================== 22:01:43 (1749607303) [ 1921.531244] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [ 1922.060763] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================================== 22:01:49 (1749607309) [ 1924.207069] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ========================================================== 22:01:51 (1749607311) [ 1926.269140] Lustre: DEBUG MARKER: == sanity test 300r: test -1 striped directory =========== 22:01:53 (1749607313) [ 1928.161445] Lustre: DEBUG MARKER: == sanity test 300s: test lfs mkdir -c without -i ======== 22:01:55 (1749607315) [ 1929.975184] Lustre: DEBUG MARKER: == sanity test 300t: test max_mdt_stripecount ============ 22:01:57 (1749607317) [ 1933.124143] Lustre: DEBUG MARKER: == sanity test 300ua: basic overstriped dir sanity test == 22:02:00 (1749607320) [ 1936.012240] Lustre: DEBUG MARKER: == sanity test 300ub: test MDT overstriping interface [ 1938.241814] Lustre: DEBUG MARKER: == sanity test 300uc: test MDT overstriping as default [ 1940.125787] Lustre: DEBUG MARKER: == sanity test 300ud: dir split ========================== 22:02:07 (1749607327) [ 2027.119118] Lustre: DEBUG MARKER: == sanity test 300ue: dir merge ========================== 22:03:34 (1749607414) [ 2070.561557] Lustre: DEBUG MARKER: == sanity test 300uf: migrate with too many local locks == 22:04:17 (1749607457) [ 2070.623526] Lustre: DEBUG MARKER: touch/create [ 2070.839891] Lustre: DEBUG MARKER: hardlinks [ 2071.088107] Lustre: DEBUG MARKER: cancel lru [ 2071.135526] Lustre: DEBUG MARKER: migrate [ 2074.333999] Lustre: DEBUG MARKER: == sanity test 300ug: migrate overstriped dirs =========== 22:04:21 (1749607461) [ 2078.081092] Lustre: DEBUG MARKER: == sanity test 300uh: overstripe tunable max_stripes_per_mdt ========================================================== 22:04:25 (1749607465) [ 2082.006293] Lustre: DEBUG MARKER: == sanity test 300ui: overstripe is not supported on one MDT system ========================================================== 22:04:29 (1749607469) [ 2082.581322] Lustre: DEBUG MARKER: SKIP: sanity test_300ui 1 MDT only [ 2083.210735] Lustre: DEBUG MARKER: == sanity test 300uj: overstriped dir with -C -N sanity test ========================================================== 22:04:30 (1749607470) [ 2091.336457] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ============= 22:04:38 (1749607478) [ 2094.701110] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open ========================================================== 22:04:42 (1749607482) [ 2097.497575] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ========================================================== 22:04:44 (1749607484) [ 2098.133246] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [ 2098.749060] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ========================================================== 22:04:46 (1749607486) [ 2120.196767] Lustre: DEBUG MARKER: == sanity test 312: make sure ZFS adjusts its block size by write pattern ========================================================== 22:05:07 (1749607507) [ 2121.460110] Lustre: DEBUG MARKER: SKIP: sanity test_312 the test only applies to zfs [ 2122.497863] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ========================================================== 22:05:09 (1749607509) [ 2127.034998] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ========================================================== 22:05:14 (1749607514) [ 2142.467981] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ============= 22:05:29 (1749607529) [ 2148.875417] Lustre: DEBUG MARKER: == sanity test 316: lfs migrate of file with large_xattr enabled ========================================================== 22:05:36 (1749607536) [ 2151.376621] Lustre: DEBUG MARKER: == sanity test 317: Verify blocks get correctly update after truncate ========================================================== 22:05:38 (1749607538) [ 2153.892177] Lustre: DEBUG MARKER: == sanity test 318: Verify async readahead tunables ====== 22:05:41 (1749607541) [ 2153.966235] LustreError: 16508:0:(lproc_llite.c:1814:read_ahead_async_file_threshold_mb_store()) lustre: can't set read_ahead_async_file_threshold_mb=65 > max_read_readahead_per_file_mb=64 [ 2155.675500] Lustre: DEBUG MARKER: == sanity test 319: lost lease lock on migrate error ===== 22:05:43 (1749607543) [ 2155.821067] LustreError: 17150:0:(ldlm_request.c:1659:ldlm_cli_cancel()) cfs_fail_timeout id 32c sleeping for 5000ms [ 2160.823969] LustreError: 17150:0:(ldlm_request.c:1659:ldlm_cli_cancel()) cfs_fail_timeout id 32c awake [ 2162.792395] Lustre: DEBUG MARKER: == sanity test 350: force NID mismatch path to be exercised ========================================================== 22:05:50 (1749607550) [ 2296.507024] Lustre: DEBUG MARKER: == sanity test 360: ldiskfs unlink in a separate thread == 22:08:04 (1749607684) [ 2308.009444] Lustre: DEBUG MARKER: == sanity test 398a: direct IO should cancel lock otherwise lockless ========================================================== 22:08:15 (1749607695) [ 2310.209438] Lustre: DEBUG MARKER: == sanity test 398b: DIO and buffer IO race ============== 22:08:17 (1749607697) [ 2362.163293] Lustre: DEBUG MARKER: == sanity test 398c: run fio to test AIO ================= 22:09:09 (1749607749) [ 2384.164030] Lustre: DEBUG MARKER: == sanity test 398d: run aiocp to verify block size > stripe size ========================================================== 22:09:31 (1749607771) [ 2396.307978] Lustre: DEBUG MARKER: == sanity test 398e: O_Direct open cleared by fcntl doesn't cause hang ========================================================== 22:09:43 (1749607783) [ 2398.799823] Lustre: DEBUG MARKER: == sanity test 398f: verify aio handles ll_direct_rw_pages errors correctly ========================================================== 22:09:46 (1749607786) [ 2403.825451] Lustre: DEBUG MARKER: == sanity test 398g: verify parallel dio async RPC submission ========================================================== 22:09:51 (1749607791) [ 2426.010587] Lustre: DEBUG MARKER: == sanity test 398h: verify correctness of read [ 2436.182098] Lustre: DEBUG MARKER: == sanity test 398i: verify parallel dio handles ll_direct_rw_pages errors correctly ========================================================== 22:10:23 (1749607823) [ 2437.503449] Lustre: *** cfs_fail_loc=1418, val=0*** [ 2439.519470] Lustre: DEBUG MARKER: == sanity test 398j: test parallel dio where stripe size > rpc_size ========================================================== 22:10:26 (1749607826) [ 2448.614988] Lustre: DEBUG MARKER: == sanity test 398k: test enospc on first stripe ========= 22:10:36 (1749607836) [ 2457.520675] Lustre: DEBUG MARKER: SKIP: sanity test_398k 7204848 > 600000 skipping out-of-space test on OST0 [ 2458.198940] Lustre: DEBUG MARKER: == sanity test 398l: test enospc on intermediate stripe/RPC ========================================================== 22:10:45 (1749607845) [ 2462.910351] Lustre: DEBUG MARKER: SKIP: sanity test_398l 7188080 > 600000 skipping out-of-space test on OST0 [ 2475.038306] Lustre: DEBUG MARKER: == sanity test 398m: test RPC failures with parallel dio ========================================================== 22:11:02 (1749607862) [ 2475.554426] LustreError: lustre-OST0000-osc-ffff8800b38f2000: operation ost_write to node 192.168.204.134@tcp failed: rc = -5 [ 2475.557554] LustreError: 2007:0:(osc_request.c:2444:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800ac6d4e00 x1834594343974272/t0(0) o4->lustre-OST0000-osc-ffff8800b38f2000@192.168.204.134@tcp:6/4 lens 488/224 e 0 to 0 dl 1749607879 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'dd.0' uid:0 gid:0 projid:0 [ 2476.258967] LustreError: lustre-OST0000-osc-ffff8800b38f2000: operation ost_write to node 192.168.204.134@tcp failed: rc = -5 [ 2476.263920] LustreError: Skipped 3 previous similar messages [ 2476.266417] LustreError: 2006:0:(osc_request.c:2444:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800ac6d4380 x1834594343976832/t0(0) o4->lustre-OST0000-osc-ffff8800b38f2000@192.168.204.134@tcp:6/4 lens 488/224 e 0 to 0 dl 1749607880 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:0 [ 2476.275933] LustreError: 2006:0:(osc_request.c:2444:osc_brw_redo_request()) Skipped 4 previous similar messages [ 2478.276068] LustreError: lustre-OST0000-osc-ffff8800b38f2000: operation ost_write to node 192.168.204.134@tcp failed: rc = -5 [ 2478.280280] LustreError: Skipped 3 previous similar messages [ 2478.282536] LustreError: 2007:0:(osc_request.c:2444:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800ac6d6d80 x1834594343977344/t0(0) o4->lustre-OST0000-osc-ffff8800b38f2000@192.168.204.134@tcp:6/4 lens 488/224 e 0 to 0 dl 1749607882 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_00_01.0' uid:0 gid:0 projid:0 [ 2478.296480] LustreError: 2007:0:(osc_request.c:2444:osc_brw_redo_request()) Skipped 3 previous similar messages [ 2481.267958] LustreError: lustre-OST0000-osc-ffff8800b38f2000: operation ost_write to node 192.168.204.134@tcp failed: rc = -5 [ 2481.273558] LustreError: Skipped 3 previous similar messages [ 2481.275542] LustreError: 2006:0:(osc_request.c:2444:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800a89df480 x1834594343977856/t0(0) o4->lustre-OST0000-osc-ffff8800b38f2000@192.168.204.134@tcp:6/4 lens 488/224 e 0 to 0 dl 1749607885 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:0 [ 2481.285013] LustreError: 2006:0:(osc_request.c:2444:osc_brw_redo_request()) Skipped 2 previous similar messages [ 2485.296499] LustreError: lustre-OST0000-osc-ffff8800b38f2000: operation ost_write to node 192.168.204.134@tcp failed: rc = -5 [ 2485.300833] LustreError: Skipped 3 previous similar messages [ 2485.302554] LustreError: 2007:0:(osc_request.c:2444:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800a89df480 x1834594343979136/t0(0) o4->lustre-OST0000-osc-ffff8800b38f2000@192.168.204.134@tcp:6/4 lens 488/224 e 0 to 0 dl 1749607889 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:0 [ 2485.313714] LustreError: 2007:0:(osc_request.c:2444:osc_brw_redo_request()) Skipped 3 previous similar messages [ 2496.282224] LustreError: lustre-OST0000-osc-ffff8800b38f2000: operation ost_write to node 192.168.204.134@tcp failed: rc = -5 [ 2496.286823] LustreError: Skipped 7 previous similar messages [ 2496.287676] LustreError: 2008:0:(osc_request.c:2444:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800a2795500 x1834594343980544/t0(0) o4->lustre-OST0000-osc-ffff8800b38f2000@192.168.204.134@tcp:6/4 lens 488/224 e 0 to 0 dl 1749607900 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_01_00.0' uid:0 gid:0 projid:0 [ 2496.287679] LustreError: 2008:0:(osc_request.c:2444:osc_brw_redo_request()) Skipped 7 previous similar messages [ 2520.320723] LustreError: lustre-OST0000-osc-ffff8800b38f2000: operation ost_write to node 192.168.204.134@tcp failed: rc = -5 [ 2520.323730] LustreError: Skipped 10 previous similar messages [ 2520.325775] LustreError: 2008:0:(osc_request.c:2444:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800a5262a00 x1834594343984512/t0(0) o4->lustre-OST0000-osc-ffff8800b38f2000@192.168.204.134@tcp:6/4 lens 488/224 e 0 to 0 dl 1749607924 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_01_00.0' uid:0 gid:0 projid:0 [ 2520.333997] LustreError: 2008:0:(osc_request.c:2444:osc_brw_redo_request()) Skipped 11 previous similar messages [ 2530.338762] LustreError: 2009:0:(osc_request.c:2599:brw_interpret()) lustre-OST0000-osc-ffff8800b38f2000: too many resent retries for object: 10737419265:8770: rc = -5 [ 2553.160120] LustreError: lustre-OST0000-osc-ffff8800b38f2000: operation ost_read to node 192.168.204.134@tcp failed: rc = -5 [ 2553.163308] LustreError: Skipped 31 previous similar messages [ 2553.164599] LustreError: 2007:0:(osc_request.c:2444:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800ac6d5880 x1834594344014720/t0(0) o3->lustre-OST0000-osc-ffff8800b38f2000@192.168.204.134@tcp:6/4 lens 488/440 e 0 to 0 dl 1749607957 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:0 [ 2553.174508] LustreError: 2007:0:(osc_request.c:2444:osc_brw_redo_request()) Skipped 27 previous similar messages [ 2587.391976] LustreError: 2009:0:(osc_request.c:2599:brw_interpret()) lustre-OST0000-osc-ffff8800b38f2000: too many resent retries for object: 10737419265:8770: rc = -5 [ 2587.396054] LustreError: 2009:0:(osc_request.c:2599:brw_interpret()) Skipped 3 previous similar messages [ 2624.207957] LustreError: lustre-OST0001-osc-ffff8800b38f2000: operation ost_write to node 192.168.204.134@tcp failed: rc = -5 [ 2624.211216] LustreError: Skipped 51 previous similar messages [ 2624.213576] LustreError: 2009:0:(osc_request.c:2444:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8800a2795f80 x1834594344036224/t0(0) o4->lustre-OST0001-osc-ffff8800b38f2000@192.168.204.134@tcp:6/4 lens 488/224 e 0 to 0 dl 1749608028 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_01_00.0' uid:0 gid:0 projid:0 [ 2624.222591] LustreError: 2009:0:(osc_request.c:2444:osc_brw_redo_request()) Skipped 47 previous similar messages [ 2643.234019] LustreError: 2009:0:(osc_request.c:2599:brw_interpret()) lustre-OST0001-osc-ffff8800b38f2000: too many resent retries for object: 11811161089:8666: rc = -5 [ 2643.239097] LustreError: 2009:0:(osc_request.c:2599:brw_interpret()) Skipped 3 previous similar messages [ 2700.344222] LustreError: 2008:0:(osc_request.c:2599:brw_interpret()) lustre-OST0001-osc-ffff8800b38f2000: too many resent retries for object: 11811161089:8666: rc = -5 [ 2700.348857] LustreError: 2008:0:(osc_request.c:2599:brw_interpret()) Skipped 2 previous similar messages [ 2703.306290] Lustre: DEBUG MARKER: == sanity test 398n: test append with parallel DIO ======= 22:14:50 (1749608090) [ 2711.759270] Lustre: DEBUG MARKER: == sanity test 398o: right kms with DIO ================== 22:14:59 (1749608099) [ 2713.546994] Lustre: DEBUG MARKER: == sanity test 398p: race aio with buffered i/o ========== 22:15:01 (1749608101) [ 2734.447494] Lustre: DEBUG MARKER: == sanity test 398q: race dio with buffered i/o ========== 22:15:21 (1749608121) [ 2776.374976] Lustre: DEBUG MARKER: == sanity test 398r: i/o error on file read ============== 22:16:03 (1749608163) [ 2776.717512] LustreError: lustre-OST0000-osc-ffff8800b38f2000: operation ost_read to node 192.168.204.134@tcp failed: rc = -5 [ 2776.720580] LustreError: Skipped 55 previous similar messages [ 2776.722394] LustreError: 2006:0:(osc_request.c:2444:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff880136bf7800 x1834594350087424/t0(0) o3->lustre-OST0000-osc-ffff8800b38f2000@192.168.204.134@tcp:6/4 lens 488/4536 e 0 to 0 dl 1749608180 ref 2 fl Interpret:RMQU/600/0 rc -5/-5 job:'cat.0' uid:0 gid:0 projid:0 [ 2776.731242] LustreError: 2006:0:(osc_request.c:2444:osc_brw_redo_request()) Skipped 47 previous similar messages [ 2831.775439] LustreError: 2006:0:(osc_request.c:2599:brw_interpret()) lustre-OST0000-osc-ffff8800b38f2000: too many resent retries for object: 10737419265:8781: rc = -5 [ 2831.780255] LustreError: 2006:0:(osc_request.c:2599:brw_interpret()) Skipped 3 previous similar messages [ 2833.364372] Lustre: DEBUG MARKER: == sanity test 398s: i/o error on mirror file read ======= 22:17:00 (1749608220) [ 2835.329457] Lustre: DEBUG MARKER: == sanity test 399a: fake write should not be slower than normal write ========================================================== 22:17:02 (1749608222) [ 2860.130471] Lustre: DEBUG MARKER: == sanity test 399b: fake read should not be slower than normal read ========================================================== 22:17:27 (1749608247) [ 2870.195174] Lustre: DEBUG MARKER: SKIP: sanity test_400a skipping excluded test 400a [ 2870.602112] Lustre: DEBUG MARKER: == sanity test 400b: packaged headers can be compiled ==== 22:17:38 (1749608258) [ 2872.123244] Lustre: DEBUG MARKER: == sanity test 401a: Verify if 'lctl list_param -R' can list parameters recursively ========================================================== 22:17:39 (1749608259) [ 2873.906209] Lustre: DEBUG MARKER: == sanity test 401aa: Verify that 'lctl list_param -p' lists the correct path names ========================================================== 22:17:41 (1749608261) [ 2875.446655] Lustre: DEBUG MARKER: == sanity test 401ab: Check that 'lctl list_param -r' lists only readable params ========================================================== 22:17:43 (1749608263) [ 2877.024133] Lustre: DEBUG MARKER: == sanity test 401ac: Check that 'lctl list_param -w' lists only writable params ========================================================== 22:17:44 (1749608264) [ 2878.594680] Lustre: DEBUG MARKER: == sanity test 401ad: Check that 'lctl list_param -wr' is conjunctive ========================================================== 22:17:46 (1749608266) [ 2880.212208] Lustre: DEBUG MARKER: == sanity test 401b: Verify 'lctl get_param' set_param' continue after error ========================================================== 22:17:47 (1749608267) [ 2881.826011] Lustre: DEBUG MARKER: == sanity test 401c: Verify 'lctl set_param' without value fails in either format. ========================================================== 22:17:49 (1749608269) [ 2883.414165] Lustre: DEBUG MARKER: == sanity test 401d: Verify 'lctl set_param' accepts values containing '=' ========================================================== 22:17:50 (1749608270) [ 2885.152557] Lustre: DEBUG MARKER: == sanity test 401db: Verify 'lctl set_param' does not add trailing '=' ========================================================== 22:17:52 (1749608272) [ 2982.527090] Lustre: DEBUG MARKER: == sanity test 401e: verify 'lctl get_param' works with NID in parameter ========================================================== 22:19:30 (1749608370) [ 2984.042652] Lustre: DEBUG MARKER: == sanity test 401f: check 'lctl list_param' doesn't follow symlinks with --no-links ========================================================== 22:19:31 (1749608371) [ 2985.651713] Lustre: DEBUG MARKER: == sanity test 401ga: check 'set_param -C' sets params upon mount ========================================================== 22:19:33 (1749608373) [ 2985.892802] LustreError: 12434:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8800b38f2000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2985.895747] LustreError: 12434:0:(lov_obd.c:784:lov_cleanup()) Skipped 3 previous similar messages [ 2985.897959] LustreError: 12434:0:(obd_class.h:479:obd_check_dev()) Device 6 not setup [ 2985.899870] LustreError: 12434:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 2985.918721] Lustre: Unmounted lustre-client [ 2985.920210] Lustre: Skipped 1 previous similar message [ 2986.020430] Lustre: Mounted lustre-client [ 2986.021583] Lustre: Skipped 1 previous similar message [ 2987.658774] Lustre: DEBUG MARKER: == sanity test 401gb: check 'set_param -d -C' removes client params ========================================================== 22:19:35 (1749608375) [ 2989.534091] Lustre: DEBUG MARKER: == sanity test 402: Return ENOENT to lod_generate_and_set_lovea ========================================================== 22:19:37 (1749608377) [ 2991.494749] Lustre: DEBUG MARKER: == sanity test 403: i_nlink should not drop to zero due to aliasing ========================================================== 22:19:39 (1749608379) [ 2991.634007] sysctl (14520): drop_caches: 2 [ 2993.347571] Lustre: DEBUG MARKER: == sanity test 404: validate manual {de}activated works properly for OSPs ========================================================== 22:19:40 (1749608380) [ 2999.091288] Lustre: DEBUG MARKER: == sanity test 405: Various layout swap lock tests ======= 22:19:46 (1749608386) [ 3001.115058] Lustre: DEBUG MARKER: SKIP: sanity test_405 layout swap does not support DOM files so far [ 3001.595987] Lustre: DEBUG MARKER: == sanity test 406: DNE support fs default striping ====== 22:19:49 (1749608389) [ 3018.030633] Lustre: DEBUG MARKER: SKIP: sanity test_407 skipping ALWAYS excluded test 407 [ 3018.469452] Lustre: DEBUG MARKER: == sanity test 408: drop_caches should not hang due to page leaks ========================================================== 22:20:06 (1749608406) [ 3018.526788] Lustre: *** cfs_fail_loc=40a, val=0*** [ 3018.528577] LustreError: 2008:0:(osc_request.c:2866:osc_build_rpc()) lustre-OST0000-osc-ffff88012a4fc800: prep_req failed: rc = -22 [ 3018.532021] LustreError: 2008:0:(osc_cache.c:2197:osc_check_rpcs()) Read request failed with -22 [ 3020.459981] bash (17241): drop_caches: 2 [ 3022.143124] Lustre: DEBUG MARKER: == sanity test 409: Large amount of cross-MDTs hard links on the same file ========================================================== 22:20:09 (1749608409) [ 3035.698776] Lustre: DEBUG MARKER: == sanity test 410: Test inode number returned from kernel thread ========================================================== 22:20:23 (1749608423) [ 3035.780601] lustre_kinode_27904: CONFIG_X86_X32 is not set [ 3035.786776] lustre_kinode_27904: inode is 144115373111771250 [ 3035.788433] lustre_kinode_27904: inode is 144115373111771250 [ 3035.789756] lustre_kinode_27904: inode numbers are identical: 144115373111771250 [ 3037.429119] Lustre: DEBUG MARKER: == sanity test 411a: Slab allocation error with cgroup does not LBUG ========================================================== 22:20:24 (1749608424) [ 3043.111932] dd invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0 [ 3043.115046] dd cpuset=/ mems_allowed=0 [ 3043.116720] CPU: 0 PID: 19756 Comm: dd Kdump: loaded Tainted: G OE ------------ 3.10.0-7.9-debug #1 [ 3043.119776] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 3043.122465] Call Trace: [ 3043.123354] [] dump_stack+0x19/0x1b [ 3043.125650] [] dump_header+0x8e/0x254 [ 3043.127820] [] ? _raw_spin_lock+0x1e/0x20 [ 3043.130287] [] ? do_raw_spin_unlock+0x49/0x90 [ 3043.132913] [] oom_kill_process+0x2cd/0x4c0 [ 3043.134578] [] mem_cgroup_oom_synchronize+0x56a/0x5a0 [ 3043.136127] [] ? mem_cgroup_charge_common+0xc0/0xc0 [ 3043.137608] [] pagefault_out_of_memory+0x14/0x90 [ 3043.139096] [] mm_fault_error+0x6a/0x157 [ 3043.140496] [] __do_page_fault+0x442/0x500 [ 3043.141841] [] trace_do_page_fault+0x56/0x170 [ 3043.143200] [] do_async_page_fault+0x22/0xf0 [ 3043.145518] [] async_page_fault+0x28/0x30 [ 3043.147251] Task in /osc_slab_alloc killed as a result of limit of /osc_slab_alloc [ 3043.150169] memory: usage 1024kB, limit 1024kB, failcnt 4555 [ 3043.152271] memory+swap: usage 1196kB, limit 9007199254740988kB, failcnt 0 [ 3043.154503] kmem: usage 72kB, limit 2048kB, failcnt 0 [ 3043.156260] Memory cgroup stats for /osc_slab_alloc: cache:948KB rss:4KB rss_huge:0KB mapped_file:0KB swap:172KB inactive_anon:0KB active_anon:0KB inactive_file:748KB active_file:200KB unevictable:0KB [ 3043.162285] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 3043.164808] [19754] 0 19754 28839 353 12 18 0 bash [ 3043.167125] [19756] 0 19756 27026 132 10 50 0 dd [ 3043.169597] Memory cgroup out of memory: Kill process 19754 (bash) score 1 or sacrifice child [ 3043.172390] Killed process 19756 (dd), UID 0, total-vm:108104kB, anon-rss:0kB, file-rss:528kB, shmem-rss:0kB [ 3044.998563] Lustre: DEBUG MARKER: SKIP: sanity test_411b skipping ALWAYS excluded test 411b [ 3045.467368] Lustre: DEBUG MARKER: == sanity test 412: mkdir on specific MDTs =============== 22:20:32 (1749608432) [ 3047.999806] Lustre: DEBUG MARKER: == sanity test 413a: QoS mkdir with 'lfs mkdir -i -1' ==== 22:20:35 (1749608435) [ 3173.096269] Lustre: DEBUG MARKER: == sanity test 413b: QoS mkdir under dir whose default LMV starting MDT offset is -1 ========================================================== 22:22:40 (1749608560) [ 3196.984859] Lustre: DEBUG MARKER: == sanity test 413c: mkdir with default LMV max inherit rr ========================================================== 22:23:04 (1749608584) [ 3221.138730] Lustre: DEBUG MARKER: == sanity test 413d: inherit ROOT default LMV ============ 22:23:28 (1749608608) [ 3226.120766] Lustre: DEBUG MARKER: == sanity test 413e: check default max-inherit value ===== 22:23:33 (1749608613) [ 3227.945287] Lustre: DEBUG MARKER: == sanity test 413f: lfs getdirstripe -D list ROOT default LMV if it's not set on dir ========================================================== 22:23:35 (1749608615) [ 3229.840556] Lustre: DEBUG MARKER: == sanity test 413g: enforce ROOT default LMV on subdir mount ========================================================== 22:23:37 (1749608617) [ 3230.002571] Lustre: Mounted lustre-client [ 3230.003941] Lustre: Skipped 1 previous similar message [ 3230.063472] LustreError: 2849:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff880083798000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 3230.066783] LustreError: 2849:0:(lov_obd.c:784:lov_cleanup()) Skipped 3 previous similar messages [ 3230.071607] LustreError: 2849:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 3230.073454] LustreError: 2849:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 3230.093706] Lustre: Unmounted lustre-client [ 3230.094716] Lustre: Skipped 1 previous similar message [ 3235.571128] Lustre: DEBUG MARKER: == sanity test 413h: don't stick to parent for round-robin dirs ========================================================== 22:23:43 (1749608623) [ 3236.414880] Lustre: DEBUG MARKER: dir=/mnt/lustre/d413h.sanity/l1/l2/l3/l4/l5: [ 3238.046414] Lustre: DEBUG MARKER: dir=/mnt/lustre/d413h.sanity/l1/l2/l3/l4/l5/d0: [ 3239.594326] Lustre: DEBUG MARKER: dir=/mnt/lustre/d413h.sanity/l1/l2/l3/l4/l5/d0/d0: [ 3241.151264] Lustre: DEBUG MARKER: dir=/mnt/lustre/d413h.sanity/l1/l2/l3/l4/l5/d0/d0/d0: [ 3242.694722] Lustre: DEBUG MARKER: dir=/mnt/lustre/d413h.sanity/l1/l2/l3/l4/l5/d0/d0/d0/d0: [ 3244.194987] Lustre: DEBUG MARKER: dir=/mnt/lustre/d413h.sanity/l1/l2/l3/l4/l5/d0/d0/d0/d0/d0: [ 3245.734872] Lustre: DEBUG MARKER: dir=/mnt/lustre/d413h.sanity/l1/l2/l3/l4/l5/d0/d0/d0/d0/d0/d0: [ 3252.500928] Lustre: DEBUG MARKER: == sanity test 413i: check default layout inheritance ==== 22:24:00 (1749608640) [ 3254.517531] Lustre: DEBUG MARKER: == sanity test 413j: set default LMV by setxattr ========= 22:24:02 (1749608642) [ 3257.332083] Lustre: DEBUG MARKER: == sanity test 413k: QoS mkdir exclude prefixes ========== 22:24:04 (1749608644) [ 3259.561357] Lustre: DEBUG MARKER: == sanity test 413z: 413 test cleanup ==================== 22:24:07 (1749608647) [ 3267.146668] Lustre: DEBUG MARKER: == sanity test 414: simulate ENOMEM in ptlrpc_register_bulk() ========================================================== 22:24:14 (1749608654) [ 3267.240172] Lustre: *** cfs_fail_loc=521, val=0*** [ 3267.242209] LustreError: 2008:0:(niobuf.c:385:ptlrpc_register_bulk()) lustre-OST0000-osc-ffff88012a4fc800: LNetMEAttach failed x1834594363522817/1: rc = -12 [ 3289.870334] LustreError: 13680:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff88012a4fc800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 3289.874302] LustreError: 13680:0:(lov_obd.c:784:lov_cleanup()) Skipped 3 previous similar messages [ 3289.880286] LustreError: 13680:0:(obd_class.h:479:obd_check_dev()) Device 6 not setup [ 3289.883327] LustreError: 13680:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3289.956702] Lustre: Unmounted lustre-client [ 3289.957772] Lustre: Skipped 1 previous similar message [ 3290.059109] Lustre: Mounted lustre-client [ 3290.060162] Lustre: Skipped 1 previous similar message [ 3291.803822] Lustre: DEBUG MARKER: == sanity test 415: lock revoke is not missing =========== 22:24:39 (1749608679) [ 3315.103059] Lustre: DEBUG MARKER: == sanity test 416: transaction start failure won't cause system hung ========================================================== 22:25:02 (1749608702) [ 3317.082243] Lustre: DEBUG MARKER: == sanity test 417: disable remote dir, striped dir and dir migration ========================================================== 22:25:04 (1749608704) [ 3322.161402] Lustre: DEBUG MARKER: == sanity test 418: df and lfs df outputs match ========== 22:25:09 (1749608709) [ 3350.901580] Lustre: DEBUG MARKER: == sanity test 419: Verify open file by name doesn't crash kernel ========================================================== 22:25:38 (1749608738) [ 3352.947084] Lustre: DEBUG MARKER: == sanity test 420: clear SGID bit on non-directories for non-members ========================================================== 22:25:40 (1749608740) [ 3355.145386] Lustre: DEBUG MARKER: == sanity test 421a: simple rm by fid ==================== 22:25:42 (1749608742) [ 3357.108941] Lustre: DEBUG MARKER: == sanity test 421b: rm by fid on open file ============== 22:25:44 (1749608744) [ 3358.973444] Lustre: DEBUG MARKER: == sanity test 421c: rm by fid against hardlinked files == 22:25:46 (1749608746) [ 3362.990367] Lustre: DEBUG MARKER: == sanity test 421d: rmfid en masse ====================== 22:25:50 (1749608750) [ 3381.009412] Lustre: DEBUG MARKER: == sanity test 421e: rmfid in DNE ======================== 22:26:08 (1749608768) [ 3385.837143] Lustre: DEBUG MARKER: == sanity test 421f: rmfid checks permissions ============ 22:26:13 (1749608773) [ 3386.351869] Lustre: Mounted lustre-client [ 3387.968947] LustreError: 26440:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff880091ad9800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 3387.972332] LustreError: 26440:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 3387.976320] LustreError: 26440:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 3387.978512] LustreError: 26440:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3387.997700] Lustre: Unmounted lustre-client [ 3388.469363] Lustre: DEBUG MARKER: == sanity test 421g: rmfid to return errors properly ===== 22:26:15 (1749608775) [ 3393.745661] Lustre: DEBUG MARKER: == sanity test 421h: rmfid with fileset mount ============ 22:26:21 (1749608781) [ 3396.495511] Lustre: DEBUG MARKER: == sanity test 422: kill a process with RPC in progress == 22:26:23 (1749608783) [ 3417.606733] Lustre: 28360:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1749608785/real 1749608785] req@ffff88012b736d80 x1834594370393984/t0(0) o101->lustre-MDT0000-mdc-ffff88008379a800@192.168.204.134@tcp:12/10 lens 576/1240 e 0 to 1 dl 1749608805 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 [ 3417.617588] Lustre: lustre-MDT0000-mdc-ffff88008379a800: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3417.621653] Lustre: Skipped 21 previous similar messages [ 3417.627910] Lustre: lustre-MDT0000-mdc-ffff88008379a800: Connection restored to (at 192.168.204.134@tcp) [ 3417.630366] Lustre: Skipped 6 previous similar messages [ 3437.631724] Lustre: 28360:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1749608805/real 1749608805] req@ffff88012b736d80 x1834594370393984/t0(0) o101->lustre-MDT0000-mdc-ffff88008379a800@192.168.204.134@tcp:12/10 lens 576/1240 e 0 to 1 dl 1749608825 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 [ 3437.640178] Lustre: lustre-MDT0000-mdc-ffff88008379a800: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3440.872384] Lustre: DEBUG MARKER: touch [ 3457.648846] Lustre: 28360:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1749608825/real 1749608825] req@ffff88012b736d80 x1834594370393984/t0(0) o101->lustre-MDT0000-mdc-ffff88008379a800@192.168.204.134@tcp:12/10 lens 576/1240 e 0 to 1 dl 1749608845 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 [ 3460.086593] Lustre: DEBUG MARKER: == sanity test 423: statfs should return a right data ==== 22:27:27 (1749608847) [ 3463.836655] Lustre: DEBUG MARKER: == sanity test 424: simulate ENOMEM in ptl_send_rpc bulk reply ME attach ========================================================== 22:27:31 (1749608851) [ 3463.977103] Lustre: *** cfs_fail_loc=522, val=0*** [ 3463.978548] LustreError: 2009:0:(niobuf.c:996:ptl_send_rpc()) LNetMEAttach failed: -12 [ 3465.750150] Lustre: DEBUG MARKER: == sanity test 425: lock count should not exceed lru size ========================================================== 22:27:33 (1749608853) [ 3477.408690] Lustre: DEBUG MARKER: == sanity test 426: splice test on Lustre ================ 22:27:44 (1749608864) [ 3479.444957] Lustre: DEBUG MARKER: == sanity test 427: Failed DNE2 update request shouldn't corrupt updatelog ========================================================== 22:27:46 (1749608866) [ 3482.703839] Lustre: lustre-MDT0001-mdc-ffff88008379a800: Connection to lustre-MDT0001 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3482.709126] Lustre: Skipped 1 previous similar message [ 3500.005752] Lustre: lustre-MDT0001-mdc-ffff88008379a800: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp) [ 3500.011228] Lustre: Skipped 2 previous similar messages [ 3500.910434] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3501.285288] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3505.108440] Lustre: DEBUG MARKER: == sanity test 428: large block size IO should not hang == 22:28:12 (1749608892) [ 3582.661000] Lustre: DEBUG MARKER: == sanity test 429: verify if opencache flag on client side does work ========================================================== 22:29:30 (1749608970) [ 3584.658711] Lustre: DEBUG MARKER: == sanity test 430a: lseek: SEEK_DATA/SEEK_HOLE basic functionality ========================================================== 22:29:32 (1749608972) [ 3588.174592] Lustre: DEBUG MARKER: == sanity test 430b: lseek: SEEK_DATA/SEEK_HOLE special cases ========================================================== 22:29:35 (1749608975) [ 3590.239659] Lustre: DEBUG MARKER: == sanity test 430c: lseek: external tools check ========= 22:29:37 (1749608977) [ 3592.040227] Lustre: DEBUG MARKER: == sanity test 431: Restart transaction for IO =========== 22:29:39 (1749608979) [ 3594.129732] bash (4468): drop_caches: 3 [ 3596.102132] Lustre: DEBUG MARKER: == sanity test 432: mv dir from outside Lustre =========== 22:29:43 (1749608983) [ 3642.324793] Lustre: DEBUG MARKER: == sanity test 433: ldlm lock cancel releases dentries and inodes ========================================================== 22:30:29 (1749609029) [ 3651.813198] Lustre: DEBUG MARKER: == sanity test 434: Client should not send RPCs for security.selinux with SElinux disabled ========================================================== 22:30:39 (1749609039) [ 3656.031759] Lustre: DEBUG MARKER: == sanity test 440: bash completion for lfs, lctl ======== 22:30:43 (1749609043) [ 3657.710541] Lustre: DEBUG MARKER: == sanity test 442: truncate vs read/write should not panic ========================================================== 22:30:45 (1749609045) [ 3658.784618] LustreError: 9004:0:(llite_lib.c:3149:ll_truncate_inode_pages_final()) cfs_fail_timeout id 1430 sleeping for 5000ms [ 3663.786756] LustreError: 9004:0:(llite_lib.c:3149:ll_truncate_inode_pages_final()) cfs_fail_timeout id 1430 awake [ 3665.551342] Lustre: DEBUG MARKER: == sanity test 460d: Check encrypt pools output ========== 22:30:53 (1749609053) [ 3667.687031] Lustre: DEBUG MARKER: == sanity test 600a: basic test for mlock()ed file ======= 22:30:55 (1749609055) [ 3668.092100] Lustre: DEBUG MARKER: SKIP: sanity test_600a This test needs vmtouch utility [ 3668.576809] Lustre: DEBUG MARKER: == sanity test 600b: mlock a file (via vmtouch) larger than max_cached_mb ========================================================== 22:30:56 (1749609056) [ 3669.006541] Lustre: DEBUG MARKER: SKIP: sanity test_600b This test needs vmtouch utility [ 3669.478660] Lustre: DEBUG MARKER: == sanity test 600c: Test I/O when mlocked page count > @max_cached_mb ========================================================== 22:30:57 (1749609057) [ 3669.897865] Lustre: DEBUG MARKER: SKIP: sanity test_600c This test needs vmtouch utility [ 3670.350973] Lustre: DEBUG MARKER: == sanity test 600d: Test I/O with limited LRU page slots (some was mlocked) ========================================================== 22:30:57 (1749609057) [ 3670.790777] Lustre: DEBUG MARKER: SKIP: sanity test_600d This test needs vmtouch utility [ 3671.274652] Lustre: DEBUG MARKER: == sanity test 801a: write barrier user interfaces and stat machine ========================================================== 22:30:58 (1749609058) [ 3702.928883] Lustre: DEBUG MARKER: == sanity test 801b: modification will be blocked by write barrier ========================================================== 22:31:30 (1749609090) [ 3705.684958] Lustre: 13099:0:(client.c:1609:after_reply()) @@@ resending request on EINPROGRESS req@ffff880091dae680 x1834594373785600/t0(0) o36->lustre-MDT0001-mdc-ffff88008379a800@192.168.204.134@tcp:12/10 lens 488/456 e 0 to 0 dl 1749609148 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 3712.964735] Lustre: DEBUG MARKER: == sanity test 801c: rescan barrier bitmap =============== 22:31:40 (1749609100) [ 3717.087895] Lustre: lustre-MDT0001-mdc-ffff88008379a800: Connection to lustre-MDT0001 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3722.511913] Lustre: DEBUG MARKER: == sanity test 802b: be able to set MDTs to readonly ===== 22:31:50 (1749609110) [ 3723.458700] Lustre: lustre-MDT0001-mdc-ffff88008379a800: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp) [ 3726.103132] Lustre: DEBUG MARKER: == sanity test 802c: be able to set OFDs to readonly ===== 22:31:53 (1749609113) [ 3729.336684] Lustre: DEBUG MARKER: == sanity test 803a: verify agent object for remote object ========================================================== 22:31:56 (1749609116) [ 3742.077854] Lustre: DEBUG MARKER: == sanity test 803b: remote object can getattr from cache ========================================================== 22:32:09 (1749609129) [ 3744.832373] Lustre: DEBUG MARKER: == sanity test 804: verify agent entry for remote entry == 22:32:12 (1749609132) [ 3755.151174] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail [ 3755.155892] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xa698cb9112352abc to 0xa698cb911241f087 [ 3755.168916] LustreError: 2005:0:(client.c:3393:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880091daca80 x1834594373804160/t25769861140(25769861140) o101->lustre-MDT0000-mdc-ffff88008379a800@192.168.204.134@tcp:12/10 lens 520/664 e 0 to 0 dl 1749609198 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 [ 3766.395076] Lustre: DEBUG MARKER: == sanity test 805: ZFS can remove from full fs ========== 22:32:33 (1749609153) [ 3767.046572] Lustre: DEBUG MARKER: SKIP: sanity test_805 ZFS specific test [ 3767.477929] Lustre: DEBUG MARKER: == sanity test 806: Verify Lazy Size on MDS ============== 22:32:35 (1749609155) [ 3775.672300] Lustre: DEBUG MARKER: == sanity test 807a: verify LSOM syncing tool ============ 22:32:43 (1749609163) [ 3778.562886] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing cancel_lru_locks osc [ 3787.413155] Lustre: DEBUG MARKER: == sanity test 807b: verify lfs somsync utility ========== 22:32:54 (1749609174) [ 3788.535903] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing cancel_lru_locks osc [ 3795.524809] Lustre: DEBUG MARKER: == sanity test 808: Check trusted.som xattr not logged in Changelogs ========================================================== 22:33:03 (1749609183) [ 3800.673711] Lustre: DEBUG MARKER: == sanity test 809: Verify no SOM xattr store for DoM-only files ========================================================== 22:33:08 (1749609188) [ 3802.384487] Lustre: DEBUG MARKER: == sanity test 810: partial page writes on ZFS (LU-11663) ========================================================== 22:33:09 (1749609189) [ 3802.439135] Lustre: *** cfs_fail_loc=411, val=0*** [ 3802.440182] Lustre: Skipped 1 previous similar message [ 3802.973980] Lustre: *** cfs_fail_loc=411, val=0*** [ 3802.975227] Lustre: Skipped 17 previous similar messages [ 3803.992156] Lustre: *** cfs_fail_loc=411, val=0*** [ 3803.994035] Lustre: Skipped 33 previous similar messages [ 3805.748701] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 3806.240461] Lustre: DEBUG MARKER: == sanity test 812a: do not drop reqs generated when imp is going to idle (LU-11951) ========================================================== 22:33:13 (1749609193) [ 3807.107909] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid 50 [ 3807.449669] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid in FULL state after 0 sec [ 3808.511971] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state CONNECTING osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid 50 [ 3821.047624] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid in CONNECTING state after 12 sec [ 3823.269448] Lustre: DEBUG MARKER: == sanity test 812b: do not drop no resend request for idle connect ========================================================== 22:33:30 (1749609210) [ 3824.289934] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid 50 [ 3824.666655] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid in FULL state after 0 sec [ 3825.822260] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state CONNECTING osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid 50 [ 3836.353688] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid in CONNECTING state after 10 sec [ 3837.519065] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid 50 [ 3851.233978] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid in IDLE state after 13 sec [ 3853.539196] Lustre: DEBUG MARKER: == sanity test 812c: idle import vs lock enqueue race ==== 22:34:00 (1749609240) [ 3854.638553] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid 50 [ 3855.067584] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid in FULL state after 0 sec [ 3865.326164] LustreError: 23026:0:(import.c:2037:ptlrpc_disconnect_and_idle_import()) cfs_race id 533 sleeping [ 3870.139089] LustreError: 29464:0:(osc_lock.c:1027:osc_lock_enqueue()) cfs_fail_race id 533 waking [ 3870.141505] LustreError: 23026:0:(import.c:2037:ptlrpc_disconnect_and_idle_import()) cfs_fail_race id 533 awake: rc=192 [ 3870.643800] LustreError: 29464:0:(osc_lock.c:1027:osc_lock_enqueue()) cfs_fail_race id 533 waking [ 3872.779590] Lustre: DEBUG MARKER: == sanity test 813: File heat verfication ================ 22:34:20 (1749609260) [ 4001.402669] Lustre: DEBUG MARKER: == sanity test 814: sparse cp works as expected (LU-12361) ========================================================== 22:36:28 (1749609388) [ 4003.225851] Lustre: DEBUG MARKER: == sanity test 815: zero byte tiny write doesn't hang (LU-12382) ========================================================== 22:36:30 (1749609390) [ 4005.071429] Lustre: DEBUG MARKER: == sanity test 816: do not reset lru_resize on idle reconnect ========================================================== 22:36:32 (1749609392) [ 4005.967350] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid 50 [ 4006.351658] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid in FULL state after 0 sec [ 4007.258393] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid 50 [ 4020.888090] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88008379a800.ost_server_uuid in IDLE state after 13 sec [ 4022.756196] Lustre: DEBUG MARKER: == sanity test 817: nfsd won't cache write lock for exec file ========================================================== 22:36:50 (1749609410) [ 4022.912130] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ 4023.044714] NFSD: starting 90-second grace period (net ffffffff81d41940) [ 4025.280808] Lustre: DEBUG MARKER: == sanity test 818: unlink with failed llog ============== 22:36:52 (1749609412) [ 4030.400404] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail [ 4030.400816] Lustre: lustre-MDT0000-mdc-ffff88008379a800: Connection to lustre-MDT0000 (at 192.168.204.134@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4030.400819] Lustre: Skipped 2 previous similar messages [ 4030.415068] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xa698cb911241f087 to 0xa698cb9112422673 [ 4030.424517] Lustre: MGC192.168.204.134@tcp: Connection restored to 192.168.204.134@tcp (at 192.168.204.134@tcp) [ 4030.430220] Lustre: Skipped 3 previous similar messages [ 4030.438154] LustreError: 2005:0:(client.c:3393:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88012dcdfb80 x1834594373929984/t30064771236(30064771236) o101->lustre-MDT0000-mdc-ffff88008379a800@192.168.204.134@tcp:12/10 lens 576/608 e 0 to 0 dl 1749609473 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'md5sum.0' uid:0 gid:0 projid:0 [ 4050.431142] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail [ 4050.436307] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xa698cb9112422673 to 0xa698cb9112422afd [ 4050.449039] LustreError: 2005:0:(client.c:3393:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88012dcdfb80 x1834594373929984/t30064771236(30064771236) o101->lustre-MDT0000-mdc-ffff88008379a800@192.168.204.134@tcp:12/10 lens 576/608 e 0 to 0 dl 1749609493 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'md5sum.0' uid:0 gid:0 projid:0 [ 4050.458811] LustreError: 2005:0:(client.c:3393:ptlrpc_replay_interpret()) Skipped 4 previous similar messages [ 4055.665337] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4056.030855] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4057.740728] Lustre: DEBUG MARKER: == sanity test 819a: too big niobuf in read ============== 22:37:25 (1749609445) [ 4059.860468] Lustre: DEBUG MARKER: == sanity test 819b: too big niobuf in write ============= 22:37:27 (1749609447) [ 4076.453801] Lustre: 2009:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1749609448/real 1749609448] req@ffff8800837c3800 x1834594374139776/t0(0) o4->lustre-OST0000-osc-ffff88008379a800@192.168.204.134@tcp:6/4 lens 488/448 e 0 to 1 dl 1749609464 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 [ 4079.283887] Lustre: DEBUG MARKER: == sanity test 820: update max EA from open intent ======= 22:37:46 (1749609466) [ 4079.518518] LustreError: 5246:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff88008379a800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4079.522689] LustreError: 5246:0:(lov_obd.c:784:lov_cleanup()) Skipped 5 previous similar messages [ 4079.527887] LustreError: 5246:0:(obd_class.h:479:obd_check_dev()) Device 6 not setup [ 4079.529736] LustreError: 5246:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 4079.551783] Lustre: Unmounted lustre-client [ 4079.553297] Lustre: Skipped 2 previous similar messages [ 4082.155289] Lustre: Mounted lustre-client [ 4082.157324] Lustre: Skipped 2 previous similar messages [ 4093.851584] LustreError: lustre-OST0001-osc-ffff880096c29800: operation ost_connect to node 192.168.204.134@tcp failed: rc = -16 [ 4093.853988] LustreError: Skipped 19 previous similar messages [ 4098.928677] Lustre: DEBUG MARKER: == sanity test 823: Setting create_count > OST_MAX_PRECREATE is lowered to maximum ========================================================== 22:38:06 (1749609486) [ 4101.028246] Lustre: DEBUG MARKER: setting create_count to 100200: [ 4101.472657] Lustre: DEBUG MARKER: -result- count: 9984 with max: 20000, expecting: 9984 [ 4103.763469] Lustre: DEBUG MARKER: == sanity test 831: throttling unlink/setattr queuing on OSP ========================================================== 22:38:11 (1749609491) [ 4109.270025] Lustre: 7804:0:(client.c:1609:after_reply()) @@@ resending request on EINPROGRESS req@ffff880087a0b100 x1834594374697728/t0(0) o36->lustre-MDT0000-mdc-ffff880096c29800@192.168.204.134@tcp:12/10 lens 488/456 e 0 to 0 dl 1749609552 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4109.276564] Lustre: 7804:0:(client.c:1609:after_reply()) Skipped 2 previous similar messages [ 4112.293308] Lustre: 7804:0:(client.c:1609:after_reply()) @@@ resending request on EINPROGRESS req@ffff880087a0ad80 x1834594374699136/t0(0) o36->lustre-MDT0000-mdc-ffff880096c29800@192.168.204.134@tcp:12/10 lens 488/456 e 0 to 0 dl 1749609555 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4118.427436] Lustre: 7804:0:(client.c:1609:after_reply()) @@@ resending request on EINPROGRESS req@ffff880096da2680 x1834594374726272/t0(0) o36->lustre-MDT0000-mdc-ffff880096c29800@192.168.204.134@tcp:12/10 lens 488/456 e 0 to 0 dl 1749609561 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4124.536599] Lustre: 7804:0:(client.c:1609:after_reply()) @@@ resending request on EINPROGRESS req@ffff88007144e680 x1834594374753408/t0(0) o36->lustre-MDT0000-mdc-ffff880096c29800@192.168.204.134@tcp:12/10 lens 488/456 e 0 to 0 dl 1749609567 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4133.771937] Lustre: 7804:0:(client.c:1609:after_reply()) @@@ resending request on EINPROGRESS req@ffff88008cd3f800 x1834594374807680/t0(0) o36->lustre-MDT0000-mdc-ffff880096c29800@192.168.204.134@tcp:12/10 lens 488/456 e 0 to 0 dl 1749609576 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4133.778695] Lustre: 7804:0:(client.c:1609:after_reply()) Skipped 2 previous similar messages [ 4152.089812] Lustre: 7804:0:(client.c:1609:after_reply()) @@@ resending request on EINPROGRESS req@ffff88012c509500 x1834594374888960/t0(0) o36->lustre-MDT0000-mdc-ffff880096c29800@192.168.204.134@tcp:12/10 lens 488/456 e 0 to 0 dl 1749609595 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4152.097810] Lustre: 7804:0:(client.c:1609:after_reply()) Skipped 2 previous similar messages [ 4170.533011] Lustre: DEBUG MARKER: == sanity test 832: lfs rm_entry ========================= 22:39:18 (1749609558) [ 4172.517608] Lustre: DEBUG MARKER: == sanity test 833: Mixed buffered/direct read and write should not return -EIO ========================================================== 22:39:19 (1749609559) [ 4206.652170] Lustre: DEBUG MARKER: SKIP: sanity test_842 skipping SLOW test 842 [ 4207.104793] Lustre: DEBUG MARKER: == sanity test 850: lljobstat can parse living and aggregated job_stats ========================================================== 22:39:54 (1749609594) [ 4209.233062] Lustre: DEBUG MARKER: == sanity test 851: fanotify can monitor open/read/write/close events for lustre fs ========================================================== 22:39:56 (1749609596) [ 4211.477502] Lustre: DEBUG MARKER: == sanity test 852: mkdir using intent lock for striped directory ========================================================== 22:39:59 (1749609599) [ 4213.362135] Lustre: DEBUG MARKER: == sanity test 853: Verify that random fadvise works as expected ========================================================== 22:40:00 (1749609600) [ 4218.676818] Lustre: DEBUG MARKER: == sanity test 900: umount should not race with any mgc requeue thread ========================================================== 22:40:06 (1749609606) [ 4237.376072] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 192.168.204.134@tcp) was lost; in progress operations using this service will fail [ 4237.382159] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xa698cb9112422f6b to 0xa698cb91124369fb [ 4237.386774] LustreError: 2005:0:(client.c:3393:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800716a2680 x1834594374944512/t38654708691(38654708691) o101->lustre-MDT0000-mdc-ffff880096c29800@192.168.204.134@tcp:12/10 lens 520/664 e 0 to 0 dl 1749609680 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 [ 4237.394061] LustreError: 2005:0:(client.c:3393:ptlrpc_replay_interpret()) Skipped 4 previous similar messages [ 4239.202319] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4239.271694] LustreError: 5354:0:(mgc_request.c:1758:mgc_process_log()) cfs_fail_timeout id 903 sleeping for 20000ms [ 4239.572366] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4259.274773] LustreError: 5354:0:(mgc_request.c:1758:mgc_process_log()) cfs_fail_timeout id 903 awake [ 4259.282278] LustreError: 5354:0:(mgc_request.c:1758:mgc_process_log()) cfs_fail_timeout id 903 sleeping for 20000ms [ 4279.284768] LustreError: 5354:0:(mgc_request.c:1758:mgc_process_log()) cfs_fail_timeout id 903 awake [ 4279.292979] LustreError: 5354:0:(mgc_request.c:1758:mgc_process_log()) cfs_fail_timeout id 903 sleeping for 20000ms [ 4299.295794] LustreError: 5354:0:(mgc_request.c:1758:mgc_process_log()) cfs_fail_timeout id 903 awake [ 4299.298571] LustreError: 5354:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4335.486826] Key type lgssc unregistered [ 4335.571986] LNet: 15003:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4336.581077] LNet: Removed LNI 192.168.204.34@tcp [ 4339.688350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 4339.692748] alg: No test for adler32 (adler32-zlib) [ 4340.621930] Lustre: Lustre: Build Version: 2.16.56_1_g45307c0 [ 4340.812436] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 4340.814084] LNet: Accept secure, port 988 [ 4342.364832] Key type lgssc registered [ 4342.765709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4530.884839] LustreError: 17363:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.134@tcp: failed processing log, type 1: rc = -5 [ 4536.903894] LustreError: 17377:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4686.886873] LustreError: MGC192.168.204.134@tcp: Confguration from log lustre-client failed from MGS -5. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4686.891216] Lustre: Unmounted lustre-client [ 4686.892481] LustreError: 17363:0:(obd_class.h:479:obd_check_dev()) Device 0 not setup [ 4686.897168] LustreError: 17363:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -5 [ 4692.955818] LustreError: 17525:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.134@tcp: failed processing log, type 1: rc = -5 [ 4700.352944] LustreError: 17539:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4848.957840] LustreError: MGC192.168.204.134@tcp: Confguration from log lustre-client failed from MGS -5. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4848.966372] Lustre: Unmounted lustre-client [ 4848.969157] LustreError: 17525:0:(obd_class.h:479:obd_check_dev()) Device 0 not setup [ 4848.972769] LustreError: 17525:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4848.976540] LustreError: 17525:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -5 [ 4851.085182] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4859.779122] Lustre: DEBUG MARKER: == sanity test 901: don't leak a mgc lock on client umount ========================================================== 22:50:47 (1749610247) [ 4860.200724] Lustre: DEBUG MARKER: sanity test_901: @@@@@@ FAIL: umount failed [ 4862.628920] Lustre: DEBUG MARKER: == sanity test 902: test short write doesn't hang lustre ========================================================== 22:50:50 (1749610250) [ 4864.521337] Lustre: DEBUG MARKER: == sanity test 903: Test long page discard does not cause evictions ========================================================== 22:50:52 (1749610252) [ 4867.970339] Lustre: DEBUG MARKER: == sanity test 904: virtual project ID xattr ============= 22:50:55 (1749610255) [ 4869.472340] Lustre: DEBUG MARKER: sanity test_904: @@@@@@ FAIL: set /mnt/lustre/d904.sanity/f904.sanity project id failed [ 4871.461374] Lustre: DEBUG MARKER: == sanity test 905: bad or new opcode should not stuck client ========================================================== 22:50:58 (1749610258) [ 4872.038164] Lustre: DEBUG MARKER: sanity test_905: @@@@@@ FAIL: setstripe failed